pub struct TenantDelta {Show 23 fields
pub lora_a: HashMap<String, Tensor>,
pub lora_b: HashMap<String, Tensor>,
pub vs: VarStore,
pub muon_states: HashMap<String, MuonState>,
pub muon_config: MuonConfig,
pub scaling_map: HashMap<String, f64>,
pub scaling: f64,
pub rank: usize,
pub device: Device,
pub target_modules: Vec<String>,
pub num_layers: usize,
pub learning_rate: f64,
pub accumulated_steps: u64,
pub max_accumulated_steps: u64,
pub request_count: u64,
pub avg_loss_improvement: f64,
pub last_access: Instant,
pub created_at: Instant,
pub last_snapshot_hash: Option<String>,
pub decay_lambda: f64,
pub effective_ranks: HashMap<String, usize>,
pub rank_oracle: Option<RankOracle>,
pub adaptation_state: DeltaAdaptationState,
/* private fields */
}Expand description
Per-tenant LoRA delta with per-layer accumulation tracking
Each tenant gets isolated per-layer A/B weight matrices. Keys use the
format "layer_idx.module_name" (e.g., "0.q_proj", "27.v_proj").
The delta accumulates across requests and can be periodically saved or
merged into a permanent adapter.
Fields§
§lora_a: HashMap<String, Tensor>LoRA A matrices: “layer_idx.module_name” -> [rank, in_features]
lora_b: HashMap<String, Tensor>LoRA B matrices: “layer_idx.module_name” -> [out_features, rank]
vs: VarStoreVarStore owning all trainable parameters
muon_states: HashMap<String, MuonState>Per-parameter Muon optimizer state: VarStore path -> MuonState
muon_config: MuonConfigMuon optimizer configuration
scaling_map: HashMap<String, f64>Per-key scaling: “layer_idx.module_name” -> alpha / layer_rank
Replaces the single scaling field for non-uniform rank support (C7 fix).
scaling: f64Default scaling factor: alpha / rank (for uniform-rank deltas or backward compat)
rank: usizeLoRA rank
device: DeviceDevice for tensors
target_modules: Vec<String>Target module names (without layer prefix)
num_layers: usizeNumber of model layers
learning_rate: f64Learning rate for optimizer
accumulated_steps: u64Total gradient steps accumulated
max_accumulated_steps: u64Maximum steps before requiring save/merge
request_count: u64Number of inference requests served
avg_loss_improvement: f64Running average of loss improvement per adaptation
last_access: InstantLast time this delta was accessed
created_at: InstantWhen this delta was created
last_snapshot_hash: Option<String>Hash of last CAS snapshot (if any)
decay_lambda: f64Weight decay factor (used by Muon’s decoupled weight decay)
effective_ranks: HashMap<String, usize>Per-key effective rank (for narrow-based rank adaptation). Key = “layer_idx.module_name”, value = active rank (1..=allocated_rank). Defaults to the allocated rank for each key.
rank_oracle: Option<RankOracle>Optional per-tenant rank oracle for runtime rank adaptation
adaptation_state: DeltaAdaptationStateLifecycle state of any pending adaptation (Idle or Pending with snapshot).
Implementations§
Source§impl TenantDelta
impl TenantDelta
Sourcepub fn new(
config: &TenantDeltaConfig,
module_dims: &HashMap<String, (usize, usize)>,
device: Device,
num_layers: usize,
) -> Result<TenantDelta, Error>
pub fn new( config: &TenantDeltaConfig, module_dims: &HashMap<String, (usize, usize)>, device: Device, num_layers: usize, ) -> Result<TenantDelta, Error>
Create a new per-layer tenant delta with Kaiming init for A and zeros for B.
Creates num_layers × num_modules A/B pairs keyed as "layer_idx.module_name".
§Arguments
config- Delta configuration (rank, alpha, target modules, layer_overrides, etc.)module_dims- Map of module_name -> (in_features, out_features) (flat, for uniform layers)device- Device for tensor allocationnum_layers- Number of model layersper_layer_dims- Optional per-layer dim overrides: layer_idx -> module_name -> (in, out) Required when different layer types have differento_projdimensions (e.g. Qwen3.5).
Sourcepub fn new_with_per_layer_dims(
config: &TenantDeltaConfig,
module_dims: &HashMap<String, (usize, usize)>,
device: Device,
num_layers: usize,
per_layer_dims: Option<&HashMap<usize, HashMap<String, (usize, usize)>>>,
) -> Result<TenantDelta, Error>
pub fn new_with_per_layer_dims( config: &TenantDeltaConfig, module_dims: &HashMap<String, (usize, usize)>, device: Device, num_layers: usize, per_layer_dims: Option<&HashMap<usize, HashMap<String, (usize, usize)>>>, ) -> Result<TenantDelta, Error>
Create a new per-layer tenant delta with optional per-layer dimension overrides.
Same as new() but accepts per_layer_dims for architectures where
o_proj (or other modules) have different dimensions across layer types.
Sourcepub fn has_module(&self, module_name: &str, layer_idx: usize) -> bool
pub fn has_module(&self, module_name: &str, layer_idx: usize) -> bool
Check if this delta has a module at the given layer
Sourcepub fn forward(
&self,
x: &Tensor,
module_name: &str,
layer_idx: usize,
) -> Result<Tensor, Error>
pub fn forward( &self, x: &Tensor, module_name: &str, layer_idx: usize, ) -> Result<Tensor, Error>
Sourcepub fn forward_2d(
&self,
x: &Tensor,
module_name: &str,
layer_idx: usize,
) -> Result<Tensor, Error>
pub fn forward_2d( &self, x: &Tensor, module_name: &str, layer_idx: usize, ) -> Result<Tensor, Error>
Compute LoRA correction for 2D tensors at a specific layer: output += scaling * (x @ A^T) @ B^T
Handles shape [batch*seq_len, features] used inside attention layers.
Multi-precision: Input x is cast to a.kind() (always Float32) via x.to_kind().
LoRA A/B are kept in Float32 regardless of the base model’s dtype. This ensures
Newton-Schulz convergence in the Muon optimizer and gradient precision within
TTT’s tight 1-5 step budget. See also muon::ns_compute_kind().
§Arguments
x- Input tensor [tokens, in_features]module_name- Name of the target module (e.g., “q_proj”, “v_proj”)layer_idx- Layer index
§Returns
LoRA correction tensor [tokens, out_features]
Sourcepub fn delta_norm_ratio(
&self,
base_norms: &HashMap<String, f64>,
) -> HashMap<String, f64>
pub fn delta_norm_ratio( &self, base_norms: &HashMap<String, f64>, ) -> HashMap<String, f64>
Compute the ratio of delta norm to a reference norm for drift monitoring
Returns a map of “layer_idx.module_name” -> ||delta|| / ||base|| where delta is the effective weight change (B @ A) and base is the reference norm.
Sourcepub fn extract_state_dict(&self) -> HashMap<String, Tensor>
pub fn extract_state_dict(&self) -> HashMap<String, Tensor>
Extract state dict (A and B tensors, effective ranks, Muon momentum) for serialization
Keys use the format "layer_idx.module_name.lora_a" / "layer_idx.module_name.lora_b".
Effective ranks are stored as "layer_idx.module_name.__effective_rank" (scalar i64).
Muon momentum buffers are stored as "layer_idx.module_name.__muon_momentum".
Sourcepub fn load_state_dict(
&mut self,
state: &HashMap<String, Tensor>,
) -> Result<(), Error>
pub fn load_state_dict( &mut self, state: &HashMap<String, Tensor>, ) -> Result<(), Error>
Load state dict (restore A/B tensors, effective ranks, and Muon momentum from a snapshot)
Expects keys in format "layer_idx.module_name.lora_a" / "layer_idx.module_name.lora_b".
Also restores __effective_rank and __muon_momentum metadata if present.
Sourcepub fn weight_epoch(&self) -> u64
pub fn weight_epoch(&self) -> u64
Current weight epoch (version counter for KV-cache staleness detection).
Sourcepub fn bump_weight_epoch(&mut self)
pub fn bump_weight_epoch(&mut self)
Bump the weight epoch. Must be called on every in-place mutation of the effective weights that happens without a remove/re-add of the delta — adaptations, train steps, rollback restores, resets. O(1); the session KV layer compares epochs on reuse and clears weight-stale caches.
Sourcepub fn memory_bytes(&self) -> usize
pub fn memory_bytes(&self) -> usize
Estimate memory usage in bytes
Sourcepub fn is_at_capacity(&self) -> bool
pub fn is_at_capacity(&self) -> bool
Check if the delta has reached its accumulation limit
Sourcepub fn effective_rank(&self, key: &str) -> Option<usize>
pub fn effective_rank(&self, key: &str) -> Option<usize>
Get the effective (active) rank for a key. Returns None if key doesn’t exist.
Sourcepub fn set_effective_rank(&mut self, key: &str, rank: usize)
pub fn set_effective_rank(&mut self, key: &str, rank: usize)
Set the effective rank for a key. Clamped to [1, max_rank]. Also updates the scaling factor for this key (alpha / effective_rank).
Source§impl TenantDelta
impl TenantDelta
Sourcepub fn compose(
base: &Arc<Mutex<RawMutex, TenantDelta>>,
tenant: &Arc<Mutex<RawMutex, TenantDelta>>,
) -> Arc<Mutex<RawMutex, TenantDelta>> ⓘ
pub fn compose( base: &Arc<Mutex<RawMutex, TenantDelta>>, tenant: &Arc<Mutex<RawMutex, TenantDelta>>, ) -> Arc<Mutex<RawMutex, TenantDelta>> ⓘ
Compose two deltas: base + tenant
Creates a new “view” delta whose forward pass sums corrections from both. The composed delta is non-trainable (for inference only).
Both deltas’ A/B matrices are summed element-wise for shared keys. For keys only in one delta, those corrections are used as-is.
Source§impl TenantDelta
impl TenantDelta
Sourcepub fn serialize_to_safetensors_bytes(&self) -> Result<Vec<u8>, Error>
pub fn serialize_to_safetensors_bytes(&self) -> Result<Vec<u8>, Error>
Serialize per-layer LoRA A/B matrices to safetensors bytes (in memory).
Uses standard HuggingFace PEFT naming convention:
base_model.model.layers.{N}.{self_attn|mlp}.{module}.lora_{A|B}.weight
Sourcepub fn load_from_safetensors_bytes(
data: &[u8],
device: Device,
) -> Result<TenantDelta, Error>
pub fn load_from_safetensors_bytes( data: &[u8], device: Device, ) -> Result<TenantDelta, Error>
Load per-layer LoRA from safetensors bytes (in memory).
Parses standard HuggingFace PEFT naming conventions with layer indices.
Trait Implementations§
impl Send for TenantDelta
impl Sync for TenantDelta
Auto Trait Implementations§
impl Freeze for TenantDelta
impl RefUnwindSafe for TenantDelta
impl Unpin for TenantDelta
impl UnsafeUnpin for TenantDelta
impl UnwindSafe for TenantDelta
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request