pub struct DeltaPool { /* private fields */ }Expand description
Registry managing per-tenant LoRA deltas for isolated TTT adaptation
Each Subject gets its own isolated TenantDelta wrapped in Arc<Mutex<_>>.
Uses DashMap for lock-free access to different tenants.
Implementations§
Source§impl DeltaPool
impl DeltaPool
Sourcepub fn new(
config: TenantDeltaConfig,
module_dims: HashMap<String, (usize, usize)>,
device: Device,
kv_registry: Option<Arc<KVCacheRegistry>>,
snapshots_dir: PathBuf,
fs: Option<WorktreeClient>,
num_layers: usize,
) -> DeltaPool
pub fn new( config: TenantDeltaConfig, module_dims: HashMap<String, (usize, usize)>, device: Device, kv_registry: Option<Arc<KVCacheRegistry>>, snapshots_dir: PathBuf, fs: Option<WorktreeClient>, num_layers: usize, ) -> DeltaPool
Create a new delta pool
§Arguments
config- Default configuration for new deltasmodule_dims- Module dimensions from model_infodevice- Device for tensor allocationkv_registry- Optional KV cache registry for dependency-aware evictionsnapshots_dir- Directory to write eviction snapshotsfs- Optional WorktreeClient for worktree-scoped file operations
Sourcepub fn with_rank_oracle(self, config: RankOracleConfig) -> DeltaPool
pub fn with_rank_oracle(self, config: RankOracleConfig) -> DeltaPool
Set rank oracle config for runtime rank adaptation on new deltas.
Sourcepub fn with_per_layer_dims(
self,
per_layer_dims: HashMap<usize, HashMap<String, (usize, usize)>>,
) -> DeltaPool
pub fn with_per_layer_dims( self, per_layer_dims: HashMap<usize, HashMap<String, (usize, usize)>>, ) -> DeltaPool
Set per-layer dimension overrides for hybrid architectures (e.g., Qwen3.5).
Sourcepub fn with_memory_budget(self, budget_bytes: usize) -> DeltaPool
pub fn with_memory_budget(self, budget_bytes: usize) -> DeltaPool
Set memory budget for the pool
Sourcepub fn with_max_tenants(self, max_tenants: usize) -> DeltaPool
pub fn with_max_tenants(self, max_tenants: usize) -> DeltaPool
Set maximum number of tenants before LRU eviction
Sourcepub fn update_config(&self, config: TenantDeltaConfig)
pub fn update_config(&self, config: TenantDeltaConfig)
Update the default config for new deltas (e.g., when createLora changes target modules)
Also clears existing deltas since their module sets would be stale.
Sourcepub fn set_base_weight_norms(&mut self, norms: HashMap<String, f64>)
pub fn set_base_weight_norms(&mut self, norms: HashMap<String, f64>)
Set base weight norms for drift monitoring
Sourcepub fn set_kv_registry(&mut self, registry: Arc<KVCacheRegistry>)
pub fn set_kv_registry(&mut self, registry: Arc<KVCacheRegistry>)
Set the KV cache registry for dependency-aware eviction
Sourcepub fn get_or_create(
&self,
tenant_id: &Subject,
) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>
pub fn get_or_create( &self, tenant_id: &Subject, ) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>
Get or create a delta for the given tenant (synchronous, no snapshot rehydration).
If the delta doesn’t exist, creates a new zero-initialized one with the default
config. This path does not reload adapters/.snapshots/{subject}.safetensors —
it cannot, because snapshot reads require async FsOps. Async callers that want
replica-loss durability (#869) must use Self::get_or_hydrate instead; this
sync entry point remains for non-async callers (tests, batched-LoRA construction).
Sourcepub async fn get_or_hydrate(
&self,
tenant_id: &Subject,
) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>
pub async fn get_or_hydrate( &self, tenant_id: &Subject, ) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>
Get an existing delta, or create one rehydrating from a persisted snapshot
if adapters/.snapshots/{subject}.safetensors exists (async sibling of
Self::get_or_create, #869).
This is the read half that makes eviction/drain snapshots durable across replica loss and scale-in. On a cold miss it:
- builds a blank delta (as
get_or_createwould), - looks for a snapshot (via FsOps when a
WorktreeClientis present, else the localsnapshots_dir), - on hit, restores the LoRA A/B weights, per-key effective ranks, Muon momentum,
and
accumulated_steps/request_countonto the delta (device-mapped to the pool’s device), - on a corrupt/unloadable/shape-incompatible snapshot, falls open to a blank
delta (the pre-#869 status quo) and quarantines the bad file as
*.corrupt.
The snapshot file is kept after a successful load (not deleted) — see the module note on snapshot lifecycle. It is overwritten by the next eviction/export.
Sourcepub fn get(
&self,
tenant_id: &Subject,
) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>
pub fn get( &self, tenant_id: &Subject, ) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>
Get an existing delta (returns None if not found)
Sourcepub fn remove(
&self,
subject: &Subject,
) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>
pub fn remove( &self, subject: &Subject, ) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>
Remove a subject’s delta and invalidate dependent KV caches
Sourcepub fn list_subjects(&self) -> Vec<String>
pub fn list_subjects(&self) -> Vec<String>
List all subjects in the pool (for debugging)
Sourcepub fn tenant_count(&self) -> usize
pub fn tenant_count(&self) -> usize
Get number of active tenants
Sourcepub fn total_memory_usage(&self) -> usize
pub fn total_memory_usage(&self) -> usize
Get total memory usage across all deltas in bytes
Sourcepub fn default_config(&self) -> TenantDeltaConfig
pub fn default_config(&self) -> TenantDeltaConfig
Get a clone of the current default config
Sourcepub fn base_weight_norms(&self) -> &HashMap<String, f64>
pub fn base_weight_norms(&self) -> &HashMap<String, f64>
Get base weight norms reference
Sourcepub fn snapshots_dir(&self) -> &PathBuf
pub fn snapshots_dir(&self) -> &PathBuf
Get snapshots directory reference
Sourcepub fn needs_eviction(&self) -> bool
pub fn needs_eviction(&self) -> bool
Check if eviction is needed (tenant count or memory budget exceeded).
This is a fast, non-blocking check with no I/O.
Sourcepub async fn ensure_capacity(&self) -> Result<(), Error>
pub async fn ensure_capacity(&self) -> Result<(), Error>
Ensure the pool has capacity for at least one more tenant.
Call this from async context before get_or_create() to avoid
Handle::block_on() panics inside spawn_local on current_thread runtime.
Evicts LRU tenants (with async snapshot I/O) until capacity is available.
Sourcepub async fn evict_lru_async(
&self,
) -> Result<Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>, Error>
pub async fn evict_lru_async( &self, ) -> Result<Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>, Error>
Async eviction of the least-recently-used tenant delta.
Before eviction:
- Snapshots the delta to a file (if it has accumulated steps) using async I/O
- Invalidates dependent KV caches (if kv_registry is set)
Returns the evicted tenant ID, delta, and optional snapshot path.
Sourcepub fn evict_lru(
&self,
) -> Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>
pub fn evict_lru( &self, ) -> Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>
Evict the least-recently-used tenant delta (sync version).
WARNING: This uses futures::executor::block_on() for snapshot I/O.
Only safe to call from non-async contexts (tests, standalone threads).
In async contexts, use evict_lru_async() instead.
Sourcepub async fn evict_to_budget_async(
&self,
) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>
pub async fn evict_to_budget_async( &self, ) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>
Async version: evict deltas until total memory is under budget.
Returns a list of evicted (tenant_id, snapshot_path) pairs for audit/recovery.
Sourcepub fn evict_to_budget(&self) -> Vec<(Subject, Option<PathBuf>)>
pub fn evict_to_budget(&self) -> Vec<(Subject, Option<PathBuf>)>
Evict deltas until total memory is under budget (sync version).
WARNING: Only safe in non-async contexts. See evict_lru() docs.
Sourcepub async fn export_all(&self) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>
pub async fn export_all(&self) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>
Drain-time export: snapshot every resident delta with accumulated steps, without evicting any of them (#869, deliverable 2).
This is the explicit “snapshot all, now” entry point for a graceful-shutdown /
pod-preStop drain hook. It reuses the eviction snapshot format and path
(adapters/.snapshots/{subject}.safetensors) and the same 512 MB cap and
FsOps-required semantics, so a rehydrating replica reads these back through
Self::get_or_hydrate identically to eviction snapshots.
Deltas remain live in the pool after export (the process is about to exit, but
the delta state must survive to disk, not be cleared). Returns (subject, snapshot_path) for each resident delta considered — None in the path slot
marks a loss event: a delta that was skipped (no accumulated steps, over the
size cap, or a write failure). Callers/operators should treat None with
accumulated_steps > 0 as data loss.
Requires a WorktreeClient; without one this is a logged no-op (FsOps-absent
skip, matching the eviction path — there is no worktree to write through).
Sourcepub fn tenant_ids(&self) -> Vec<Subject>
pub fn tenant_ids(&self) -> Vec<Subject>
List all tenant IDs
Auto Trait Implementations§
impl !Freeze for DeltaPool
impl !RefUnwindSafe for DeltaPool
impl Send for DeltaPool
impl Sync for DeltaPool
impl Unpin for DeltaPool
impl UnsafeUnpin for DeltaPool
impl !UnwindSafe for DeltaPool
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request