Skip to main content

DeltaPool

Struct DeltaPool 

Source
pub struct DeltaPool { /* private fields */ }
Expand description

Registry managing per-tenant LoRA deltas for isolated TTT adaptation

Each Subject gets its own isolated TenantDelta wrapped in Arc<Mutex<_>>. Uses DashMap for lock-free access to different tenants.

Implementations§

Source§

impl DeltaPool

Source

pub fn new( config: TenantDeltaConfig, module_dims: HashMap<String, (usize, usize)>, device: Device, kv_registry: Option<Arc<KVCacheRegistry>>, snapshots_dir: PathBuf, fs: Option<WorktreeClient>, num_layers: usize, ) -> DeltaPool

Create a new delta pool

§Arguments
  • config - Default configuration for new deltas
  • module_dims - Module dimensions from model_info
  • device - Device for tensor allocation
  • kv_registry - Optional KV cache registry for dependency-aware eviction
  • snapshots_dir - Directory to write eviction snapshots
  • fs - Optional WorktreeClient for worktree-scoped file operations
Source

pub fn with_rank_oracle(self, config: RankOracleConfig) -> DeltaPool

Set rank oracle config for runtime rank adaptation on new deltas.

Source

pub fn with_per_layer_dims( self, per_layer_dims: HashMap<usize, HashMap<String, (usize, usize)>>, ) -> DeltaPool

Set per-layer dimension overrides for hybrid architectures (e.g., Qwen3.5).

Source

pub fn with_memory_budget(self, budget_bytes: usize) -> DeltaPool

Set memory budget for the pool

Source

pub fn with_max_tenants(self, max_tenants: usize) -> DeltaPool

Set maximum number of tenants before LRU eviction

Source

pub fn update_config(&self, config: TenantDeltaConfig)

Update the default config for new deltas (e.g., when createLora changes target modules)

Also clears existing deltas since their module sets would be stale.

Source

pub fn set_base_weight_norms(&mut self, norms: HashMap<String, f64>)

Set base weight norms for drift monitoring

Source

pub fn set_kv_registry(&mut self, registry: Arc<KVCacheRegistry>)

Set the KV cache registry for dependency-aware eviction

Source

pub fn get_or_create( &self, tenant_id: &Subject, ) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>

Get or create a delta for the given tenant (synchronous, no snapshot rehydration).

If the delta doesn’t exist, creates a new zero-initialized one with the default config. This path does not reload adapters/.snapshots/{subject}.safetensors — it cannot, because snapshot reads require async FsOps. Async callers that want replica-loss durability (#869) must use Self::get_or_hydrate instead; this sync entry point remains for non-async callers (tests, batched-LoRA construction).

Source

pub async fn get_or_hydrate( &self, tenant_id: &Subject, ) -> Result<Arc<Mutex<RawMutex, TenantDelta>>, Error>

Get an existing delta, or create one rehydrating from a persisted snapshot if adapters/.snapshots/{subject}.safetensors exists (async sibling of Self::get_or_create, #869).

This is the read half that makes eviction/drain snapshots durable across replica loss and scale-in. On a cold miss it:

  1. builds a blank delta (as get_or_create would),
  2. looks for a snapshot (via FsOps when a WorktreeClient is present, else the local snapshots_dir),
  3. on hit, restores the LoRA A/B weights, per-key effective ranks, Muon momentum, and accumulated_steps / request_count onto the delta (device-mapped to the pool’s device),
  4. on a corrupt/unloadable/shape-incompatible snapshot, falls open to a blank delta (the pre-#869 status quo) and quarantines the bad file as *.corrupt.

The snapshot file is kept after a successful load (not deleted) — see the module note on snapshot lifecycle. It is overwritten by the next eviction/export.

Source

pub fn get( &self, tenant_id: &Subject, ) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>

Get an existing delta (returns None if not found)

Source

pub fn remove( &self, subject: &Subject, ) -> Option<Arc<Mutex<RawMutex, TenantDelta>>>

Remove a subject’s delta and invalidate dependent KV caches

Source

pub fn list_subjects(&self) -> Vec<String>

List all subjects in the pool (for debugging)

Source

pub fn tenant_count(&self) -> usize

Get number of active tenants

Source

pub fn total_memory_usage(&self) -> usize

Get total memory usage across all deltas in bytes

Source

pub fn default_config(&self) -> TenantDeltaConfig

Get a clone of the current default config

Source

pub fn rank(&self) -> usize

Get the LoRA rank used by this pool’s deltas

Source

pub fn module_dims(&self) -> &HashMap<String, (usize, usize)>

Get module dimensions reference

Source

pub fn base_weight_norms(&self) -> &HashMap<String, f64>

Get base weight norms reference

Source

pub fn snapshots_dir(&self) -> &PathBuf

Get snapshots directory reference

Source

pub fn needs_eviction(&self) -> bool

Check if eviction is needed (tenant count or memory budget exceeded).

This is a fast, non-blocking check with no I/O.

Source

pub async fn ensure_capacity(&self) -> Result<(), Error>

Ensure the pool has capacity for at least one more tenant.

Call this from async context before get_or_create() to avoid Handle::block_on() panics inside spawn_local on current_thread runtime.

Evicts LRU tenants (with async snapshot I/O) until capacity is available.

Source

pub async fn evict_lru_async( &self, ) -> Result<Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>, Error>

Async eviction of the least-recently-used tenant delta.

Before eviction:

  1. Snapshots the delta to a file (if it has accumulated steps) using async I/O
  2. Invalidates dependent KV caches (if kv_registry is set)

Returns the evicted tenant ID, delta, and optional snapshot path.

Source

pub fn evict_lru( &self, ) -> Option<(Subject, Arc<Mutex<RawMutex, TenantDelta>>, Option<PathBuf>)>

Evict the least-recently-used tenant delta (sync version).

WARNING: This uses futures::executor::block_on() for snapshot I/O. Only safe to call from non-async contexts (tests, standalone threads). In async contexts, use evict_lru_async() instead.

Source

pub async fn evict_to_budget_async( &self, ) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>

Async version: evict deltas until total memory is under budget.

Returns a list of evicted (tenant_id, snapshot_path) pairs for audit/recovery.

Source

pub fn evict_to_budget(&self) -> Vec<(Subject, Option<PathBuf>)>

Evict deltas until total memory is under budget (sync version).

WARNING: Only safe in non-async contexts. See evict_lru() docs.

Source

pub async fn export_all(&self) -> Result<Vec<(Subject, Option<PathBuf>)>, Error>

Drain-time export: snapshot every resident delta with accumulated steps, without evicting any of them (#869, deliverable 2).

This is the explicit “snapshot all, now” entry point for a graceful-shutdown / pod-preStop drain hook. It reuses the eviction snapshot format and path (adapters/.snapshots/{subject}.safetensors) and the same 512 MB cap and FsOps-required semantics, so a rehydrating replica reads these back through Self::get_or_hydrate identically to eviction snapshots.

Deltas remain live in the pool after export (the process is about to exit, but the delta state must survive to disk, not be cleared). Returns (subject, snapshot_path) for each resident delta considered — None in the path slot marks a loss event: a delta that was skipped (no accumulated steps, over the size cap, or a write failure). Callers/operators should treat None with accumulated_steps > 0 as data loss.

Requires a WorktreeClient; without one this is a logged no-op (FsOps-absent skip, matching the eviction path — there is no worktree to write through).

Source

pub fn clear_all(&self)

Clear all deltas and invalidate all dependent KV caches

Source

pub fn tenant_ids(&self) -> Vec<Subject>

List all tenant IDs

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<T> AnySync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

§

impl<'a, T, E> AsTaggedExplicit<'a, E> for T
where T: 'a,

§

fn explicit(self, class: Class, tag: u32) -> TaggedParser<'a, Explicit, Self, E>

§

impl<'a, T, E> AsTaggedImplicit<'a, E> for T
where T: 'a,

§

fn implicit( self, class: Class, constructed: bool, tag: u32, ) -> TaggedParser<'a, Implicit, Self, E>

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<T> JaggedSlice<T> for T

§

type Index = Index2

§

fn push_into(self, array: &mut Jagged<T>)

§

fn insert_into(self, index: <T as JaggedSlice<T>>::Index, array: &mut Jagged<T>)

§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> MaybeSync for T
where T: Sync,