Expand description
Per-tenant delta registry with LRU eviction
Manages a pool of TenantDelta instances, one per tenant/session.
Pattern mirrors KVCacheRegistry using DashMap for concurrent access.
§Snapshot durability & lifecycle (#869)
Uncommitted per-tenant TTT adaptation is persisted to
adapters/.snapshots/{subject}.safetensors on two occasions:
- LRU eviction (
DeltaPool::evict_lru/DeltaPool::evict_lru_async) — a warm-cache spill. - Drain-time export (
DeltaPool::export_all) — a graceful-shutdown “snapshot all, now” for pod preStop / scale-in.
The read half is DeltaPool::get_or_hydrate, which reloads a snapshot before
zero-initializing a fresh delta, restoring weights, effective ranks, Muon momentum,
and accumulated_steps / request_count.
Lifecycle decision — keep, don’t delete on load. A rehydrated snapshot is left
on disk after a successful load. Deleting it on load would open a window where a crash
between hydrate and the next eviction/export loses every step accumulated since — the
exact failure #869 exists to close. The file is instead overwritten by the next
eviction/export snapshot of the same subject, so at worst a reader sees slightly stale
(never absent) state. These staging snapshots stay in .snapshots/ and are never
auto-promoted into a committed adapters/NN_*.safetensors adapter — that crossing is
the STEP Promote boundary and remains an explicit operation.
What a snapshot does NOT preserve (rehydrated deltas start blank for these; see
TenantDelta::extract_state_dict): the in-memory STEP Pending rollback state
(DeltaAdaptationState::Pending) — a speculative, not-yet-resolved adaptation is
folded into the committed weights it was applied on top of, so it survives as
committed rather than as a reversible pending. No SSM/optimizer state beyond Muon
momentum is carried (there is none in TenantDelta today). avg_loss_improvement
and last_snapshot_hash reset.
Structs§
- Delta
Pool - Registry managing per-tenant LoRA deltas for isolated TTT adaptation