pub struct BlockPool { /* private fields */ }Expand description
Shared pool of fixed-size GPU tensor blocks for KV cache storage.
All KV cache memory is pre-allocated in this pool at startup. Sessions
allocate and free blocks from the pool — no per-session cudaMalloc calls,
zero fragmentation, and instant block reuse across sessions.
Each block stores BLOCK_SIZE tokens of K or V for a single attention layer.
Shape: [1, BLOCK_SIZE, num_kv_heads, head_dim].
Implementations§
Source§impl BlockPool
impl BlockPool
Sourcepub fn new(
num_blocks: usize,
num_kv_heads: usize,
head_dim: usize,
device: Device,
dtype: DType,
) -> Result<Self>
pub fn new( num_blocks: usize, num_kv_heads: usize, head_dim: usize, device: Device, dtype: DType, ) -> Result<Self>
Create a new block pool with num_blocks pre-allocated blocks.
All blocks are allocated on device with dtype dtype.
num_kv_heads and head_dim define the per-head tensor dimensions.
Sourcepub fn allocate(
&mut self,
owner: &CacheOwner,
) -> Result<BlockId, BlockPoolExhausted>
pub fn allocate( &mut self, owner: &CacheOwner, ) -> Result<BlockId, BlockPoolExhausted>
Allocate a block from the pool on behalf of owner.
Returns BlockPoolExhausted (a fail-soft signal a scheduler can turn
into admission-defer) when no free blocks remain. The allocated block is
tagged with owner so it can only be freed/used by that same owner.
Sourcepub fn free(&mut self, owner: &CacheOwner, id: BlockId)
pub fn free(&mut self, owner: &CacheOwner, id: BlockId)
Return a block to the pool.
Panics (in debug) / logs and refuses (in release) if owner does not
match the owner the block was allocated to — this enforces tenant
isolation: one CacheOwner can never free another’s block.
Sourcepub fn owner_of(&self, id: BlockId) -> Option<&CacheOwner>
pub fn owner_of(&self, id: BlockId) -> Option<&CacheOwner>
Owner currently holding id, if any (for tests/metrics).
Sourcepub fn used_blocks(&self) -> usize
pub fn used_blocks(&self) -> usize
Number of blocks currently in use.
Sourcepub fn free_block_count(&self) -> usize
pub fn free_block_count(&self) -> usize
Number of blocks available for allocation.
Sourcepub fn total_blocks(&self) -> usize
pub fn total_blocks(&self) -> usize
Total number of blocks in the pool.
Sourcepub fn used_memory(&self) -> usize
pub fn used_memory(&self) -> usize
Memory used by allocated blocks (approximate, in bytes).
Sourcepub fn block_shape(&self) -> &[i64; 4]
pub fn block_shape(&self) -> &[i64; 4]
Block shape: [1, BLOCK_SIZE, num_kv_heads, head_dim]
Trait Implementations§
Auto Trait Implementations§
impl Freeze for BlockPool
impl RefUnwindSafe for BlockPool
impl Send for BlockPool
impl !Sync for BlockPool
impl Unpin for BlockPool
impl UnsafeUnpin for BlockPool
impl UnwindSafe for BlockPool
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more