pub struct KVCacheManager { /* private fields */ }Expand description
KV cache manager for all layers in a model
Implementations§
Source§impl KVCacheManager
impl KVCacheManager
Sourcepub fn new(
num_layers: usize,
max_seq_len: usize,
quant_type: KVQuantType,
) -> Self
pub fn new( num_layers: usize, max_seq_len: usize, quant_type: KVQuantType, ) -> Self
Create a new KV cache manager
Sourcepub fn new_paged(
num_layers: usize,
max_seq_len: usize,
pool: Arc<Mutex<BlockPool>>,
owner: CacheOwner,
) -> Self
pub fn new_paged( num_layers: usize, max_seq_len: usize, pool: Arc<Mutex<BlockPool>>, owner: CacheOwner, ) -> Self
Create a new KV cache manager using paged block storage.
All layers share the same BlockPool for zero-fragmentation memory management.
Sourcepub fn with_layer_cache<F, R>(&self, layer_idx: usize, f: F) -> Option<R>where
F: FnOnce(&mut LayerKVCache) -> R,
pub fn with_layer_cache<F, R>(&self, layer_idx: usize, f: F) -> Option<R>where
F: FnOnce(&mut LayerKVCache) -> R,
Get cache for a specific layer with a closure
Sourcepub fn layer_cache_ref(
&self,
layer_idx: usize,
) -> Option<RefMut<'_, usize, LayerKVCache>>
pub fn layer_cache_ref( &self, layer_idx: usize, ) -> Option<RefMut<'_, usize, LayerKVCache>>
Borrow a layer’s cache directly as a RefMut (for the batched decode path,
which needs to hold one mutable layer-cache borrow per sequence across a
batched attention call). Each sequence has its own manager, so the borrows
never alias. Returns None when caching is disabled or the layer is absent.
Sourcepub fn clear_all(&mut self)
pub fn clear_all(&mut self)
Clear all KV tensors together with the metadata that describes them.
cached_token_ids, the SSM snapshot, and its slot-device record only
have meaning alongside the layer tensors they were computed with. A
cleared cache that retained them would let a later extending prompt
claim a prefix_len — or restore a stale snapshot — against KV that
no longer exists (e.g. a session turn cancelled before its first
prefill: the set-up clear runs at generation setup, and the drop
records nothing new).
Sourcepub fn ensure_layer_cache(&self, layer_idx: usize)
pub fn ensure_layer_cache(&self, layer_idx: usize)
Insert a fresh cache for layer_idx if none exists yet.
Architectures with auxiliary KV-consuming modules beyond the main
decoder layers (the Qwen3.5 MTP draft head, whose slot sits at index
num_hidden_layers) call this when a session cache is swapped in: the
registry may have created the manager with only the main layer count.
Sourcepub fn max_seq_len(&self) -> usize
pub fn max_seq_len(&self) -> usize
Maximum sequence length every layer cache is bounded by
(LayerKVCache::update errors past it).
Sourcepub fn compat_fingerprint(&self) -> Option<KvCompatFingerprint>
pub fn compat_fingerprint(&self) -> Option<KvCompatFingerprint>
The compatibility fingerprint stamped on this cache, or None until the
cache is first populated (#1277).
Sourcepub fn set_compat_fingerprint(&mut self, fp: KvCompatFingerprint)
pub fn set_compat_fingerprint(&mut self, fp: KvCompatFingerprint)
Stamp (or re-stamp) the compatibility fingerprint under which this cache’s KV is being produced. Called by the engine at reuse time: a fresh cache is stamped with the current config; an incompatible cache is cleared and re-stamped after its stale KV is discarded.
Sourcepub fn apply_reuse_policy(
&mut self,
expected: Option<KvCompatFingerprint>,
) -> CacheReuseDecision
pub fn apply_reuse_policy( &mut self, expected: Option<KvCompatFingerprint>, ) -> CacheReuseDecision
Apply the fail-closed KV compatibility policy to this real cache.
This is the production state-transition boundary for #1277. A compatible stamped cache is retained. A fresh empty cache is stamped before prefill. Every incompatible or unproven cache has both its token IDs and layer tensors discarded, then is re-stamped only when an authoritative expected fingerprint is available.
Sourcepub fn set_enabled(&mut self, enabled: bool)
pub fn set_enabled(&mut self, enabled: bool)
Enable or disable caching
Sourcepub fn memory_usage(&self) -> usize
pub fn memory_usage(&self) -> usize
Get total memory usage in bytes (approximate)
Sourcepub fn quant_type(&self) -> KVQuantType
pub fn quant_type(&self) -> KVQuantType
Get the quantization type
Sourcepub fn last_access(&self) -> u64
pub fn last_access(&self) -> u64
Get last access timestamp in milliseconds
Sourcepub fn access_count(&self) -> u64
pub fn access_count(&self) -> u64
Get access count
Sourcepub fn prefix_match_len(&self, new_tokens: &[i64]) -> usize
pub fn prefix_match_len(&self, new_tokens: &[i64]) -> usize
Find the length of the common prefix between cached tokens and new tokens.
Returns the number of tokens whose KV values can be reused from the cache. A return value of 0 means no prefix match (full recomputation needed).
Sourcepub fn set_cached_tokens(&mut self, tokens: Vec<i64>)
pub fn set_cached_tokens(&mut self, tokens: Vec<i64>)
Record which tokens this cache was computed for.
Called after generation completes so the next turn can detect prefix overlap. Any prior SSM snapshot is dropped: tokens recorded without a known recurrent state must not resurrect a stale one.
Sourcepub fn set_cached_tokens_with_ssm(
&mut self,
tokens: Vec<i64>,
ssm_snapshot: Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>,
)
pub fn set_cached_tokens_with_ssm( &mut self, tokens: Vec<i64>, ssm_snapshot: Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>, )
Record which tokens this cache was computed for, together with the
recurrent-state (SSM conv/rec) snapshot as of the end of the prefill
that produced them (hybrid models only; None for pure attention).
The snapshot is what makes a later prefix hit rewindable for recurrent layers: KV truncates to the matched prefix, the snapshot restores the GDN conv/rec state to the end of the cached sequence.
Sourcepub fn ssm_snapshot(&self) -> Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>
pub fn ssm_snapshot(&self) -> Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>
Deep-copied SSM snapshot for restore on a prefix hit.
Returns deep copies (same .copy() discipline as
Qwen3_5Model::snapshot_ssm_states) so that restoring into the live
model state never aliases — and cannot mutate — the stored snapshot.
Sourcepub fn cached_token_count(&self) -> usize
pub fn cached_token_count(&self) -> usize
Get the cached token IDs (for debugging/metrics).
Sourcepub fn truncate_to(&self, pos: usize)
pub fn truncate_to(&self, pos: usize)
Truncate the cache to a given token position.
Keeps KV values for tokens 0..pos and discards everything after. Used when a prefix matches but the suffix has changed (new turn).
Sourcepub fn location(&self) -> CacheLocation
pub fn location(&self) -> CacheLocation
Current storage location of this cache’s tensors.
Sourcepub fn offload_to_cpu(&mut self)
pub fn offload_to_cpu(&mut self)
Offload all layer caches from GPU to CPU RAM.
Frees GPU memory while preserving the cached data for later restore. Only works for FullPrecision storage — Paged caches free blocks to the pool instead (handled by evict_to_budget), and Quantized data is already partially on CPU.
Sourcepub fn restore_to_gpu(&mut self, device: Device)
pub fn restore_to_gpu(&mut self, device: Device)
Restore all layer caches from CPU back to GPU.
Must be called before using the cache for inference.
Auto Trait Implementations§
impl !Freeze for KVCacheManager
impl !RefUnwindSafe for KVCacheManager
impl Send for KVCacheManager
impl !Sync for KVCacheManager
impl Unpin for KVCacheManager
impl UnsafeUnpin for KVCacheManager
impl !UnwindSafe for KVCacheManager
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more