pub struct KVCacheRegistry { /* private fields */ }Expand description
Registry managing multiple KV cache instances for concurrent sessions.
Each CacheOwner gets their own isolated KVCacheManager wrapped in a Mutex.
The registry uses DashMap for lock-free access to different sessions, while
each individual cache uses Mutex for thread-safe layer access (required because
tch-rs Tensor contains raw pointers that aren’t Sync).
§Concurrency Model
Request A ──► Registry.get_or_create(Session("abc")) ──► Cache A (Mutex)
Request B ──► Registry.get_or_create(Session("xyz")) ──► Cache B (Mutex)
↑
No contention between A and B!Implementations§
Source§impl KVCacheRegistry
impl KVCacheRegistry
Sourcepub fn new(
default_config: CacheConfig,
memory_budget: Option<usize>,
) -> KVCacheRegistry
pub fn new( default_config: CacheConfig, memory_budget: Option<usize>, ) -> KVCacheRegistry
Create a new KV cache registry
Sourcepub fn init_block_pool(
&mut self,
num_blocks: usize,
num_kv_heads: usize,
head_dim: usize,
device: Device,
dtype: Kind,
) -> Result<(), Error>
pub fn init_block_pool( &mut self, num_blocks: usize, num_kv_heads: usize, head_dim: usize, device: Device, dtype: Kind, ) -> Result<(), Error>
Initialize the shared block pool for paged KV cache storage.
Must be called before any paged caches are created. num_blocks determines
the total KV cache capacity across all sessions. num_kv_heads and head_dim
must match the model’s attention configuration.
Sourcepub fn block_pool(&self) -> Option<&Arc<Mutex<RawMutex, BlockPool>>>
pub fn block_pool(&self) -> Option<&Arc<Mutex<RawMutex, BlockPool>>>
Get the shared block pool (for metrics/debugging).
Sourcepub fn get_or_create(
&self,
owner: CacheOwner,
) -> Arc<Mutex<RawMutex, KVCacheManager>> ⓘ
pub fn get_or_create( &self, owner: CacheOwner, ) -> Arc<Mutex<RawMutex, KVCacheManager>> ⓘ
Get or create a cache for the given owner.
If the cache doesn’t exist, creates a new one with the default config.
Returns Arc<Mutex<KVCacheManager>> - caller must lock() to use.
Sourcepub fn release(&self, owner: &CacheOwner)
pub fn release(&self, owner: &CacheOwner)
Release a cache when the session is done.
For Stateless owners, this removes the cache immediately.
For Session owners, the cache is kept for potential reuse.
Sourcepub fn evict_to_budget(&self)
pub fn evict_to_budget(&self)
Evict least-recently-used caches to stay within memory budget.
Sourcepub fn total_memory_usage(&self) -> usize
pub fn total_memory_usage(&self) -> usize
Get total memory usage across all caches in bytes
Sourcepub fn cache_count(&self) -> usize
pub fn cache_count(&self) -> usize
Get number of active caches
Sourcepub fn register_delta_dependency(
&self,
owner: &CacheOwner,
tenant_id: Option<String>,
weight_epoch: u64,
) -> bool
pub fn register_delta_dependency( &self, owner: &CacheOwner, tenant_id: Option<String>, weight_epoch: u64, ) -> bool
Register the tenant delta that produced a cache owner’s KV.
Records (tenant_id, weight_epoch) as the effective tenant whose delta
contributed to the cache’s weights, along with that delta’s weight epoch
at compute time — pass None when no tenant delta contributes (bare
base model or base-delta-only inference; the epoch is then irrelevant).
When that tenant’s delta is later evicted, reset, or removed,
Self::invalidate_for_tenant clears every cache recorded under it.
Stale-weight guard: if the effective (tenant_id, weight_epoch)
differs from the value recorded the last time this owner’s cache was
computed, the existing KV was produced under different weights and is
cleared in place (cached token IDs + every layer cache) before the new
dependency is recorded, so generation can never reuse it. Returns true
when such a stale cache was cleared. This is the barrier that stops a
session cache from crossing a weight boundary — no-delta↔delta,
tenant-A↔tenant-B, and same-tenant in-place re-adaptation (the epoch
advances while the tenant id stays the same): the next request
re-registers with the now-effective tenant and epoch, the mismatch is
detected, and the stale KV is dropped, forcing a full recompute. Both
the bump (at adapt/train/rollback/reset time) and this comparison are
O(1) — no per-adaptation scan of the dependency map. A cache with no
prior record (fresh, or just invalidated) is never treated as stale.
Sourcepub fn invalidate_for_tenant(&self, tenant_id: &str) -> usize
pub fn invalidate_for_tenant(&self, tenant_id: &str) -> usize
Invalidate all KV caches that depend on a specific subject’s delta.
Called when a subject’s delta is evicted, reset, or significantly modified. Returns the number of caches invalidated.
Auto Trait Implementations§
impl Freeze for KVCacheRegistry
impl !RefUnwindSafe for KVCacheRegistry
impl Send for KVCacheRegistry
impl Sync for KVCacheRegistry
impl Unpin for KVCacheRegistry
impl UnsafeUnpin for KVCacheRegistry
impl !UnwindSafe for KVCacheRegistry
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request