pub struct LayerKVCache {
pub seq_pos: usize,
pub max_seq_len: usize,
/* private fields */
}Expand description
KV cache for a single attention layer
Fields§
§seq_pos: usizeCurrent sequence position (number of cached tokens)
max_seq_len: usizeMaximum sequence length
Implementations§
Source§impl LayerKVCache
impl LayerKVCache
Sourcepub fn new(max_seq_len: usize, quant_type: KVQuantType) -> LayerKVCache
pub fn new(max_seq_len: usize, quant_type: KVQuantType) -> LayerKVCache
Create a new layer KV cache (without bnb feature - quantization disabled)
Sourcepub fn new_paged(
max_seq_len: usize,
pool: Arc<Mutex<RawMutex, BlockPool>>,
owner: CacheOwner,
) -> LayerKVCache
pub fn new_paged( max_seq_len: usize, pool: Arc<Mutex<RawMutex, BlockPool>>, owner: CacheOwner, ) -> LayerKVCache
Create a new layer KV cache using paged block storage owned by owner.
Sourcepub fn update(
&mut self,
new_keys: &Tensor,
new_values: &Tensor,
start_pos: usize,
) -> Result<(), Error>
pub fn update( &mut self, new_keys: &Tensor, new_values: &Tensor, start_pos: usize, ) -> Result<(), Error>
Update cache with new keys and values
Sourcepub fn get(&mut self) -> Result<(Tensor, Tensor), Error>
pub fn get(&mut self) -> Result<(Tensor, Tensor), Error>
Get cached keys and values up to current position
Sourcepub fn truncate_to(&mut self, pos: usize)
pub fn truncate_to(&mut self, pos: usize)
Truncate cache to keep only tokens at positions 0..pos.
For FullPrecision and Paged, this adjusts position markers — allocated capacity/blocks remain and will be reused. For Paged, excess blocks beyond the last needed block are freed back to the pool.
For Quantized, this falls back to a full clear since truncation within quantized blocks is complex.
Sourcepub fn is_initialized(&self) -> bool
pub fn is_initialized(&self) -> bool
Check if cache is initialized
Sourcepub fn memory_usage(&self) -> usize
pub fn memory_usage(&self) -> usize
Get memory usage in bytes (approximate)
Sourcepub fn to_device(&mut self, device: Device)
pub fn to_device(&mut self, device: Device)
Move tensors to a different device (for CPU offload/restore).
For FullPrecision: moves K/V tensors via Tensor::to_device().
For Paged: no-op (blocks are managed by the pool).
For Quantized: no-op (quantized data is already on CPU).
Sourcepub fn quant_type(&self) -> KVQuantType
pub fn quant_type(&self) -> KVQuantType
Get the quantization type
Trait Implementations§
Source§impl Debug for LayerKVCache
impl Debug for LayerKVCache
Source§impl Drop for LayerKVCache
impl Drop for LayerKVCache
Source§fn drop(&mut self)
fn drop(&mut self)
Return paged blocks to the shared pool when the cache is dropped.
Without this, registry eviction (caches.remove) would drop paged caches
without returning their blocks, leaking pool capacity until restart.
Non-paged storage owns its own tensors and needs no special handling.
Auto Trait Implementations§
impl Freeze for LayerKVCache
impl !RefUnwindSafe for LayerKVCache
impl Send for LayerKVCache
impl !Sync for LayerKVCache
impl Unpin for LayerKVCache
impl UnsafeUnpin for LayerKVCache
impl !UnwindSafe for LayerKVCache
Blanket Implementations§
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request