Skip to main content

TenantDelta

Struct TenantDelta 

Source
pub struct TenantDelta {
Show 23 fields pub lora_a: HashMap<String, Tensor>, pub lora_b: HashMap<String, Tensor>, pub vs: VarStore, pub muon_states: HashMap<String, MuonState>, pub muon_config: MuonConfig, pub scaling_map: HashMap<String, f64>, pub scaling: f64, pub rank: usize, pub device: Device, pub target_modules: Vec<String>, pub num_layers: usize, pub learning_rate: f64, pub accumulated_steps: u64, pub max_accumulated_steps: u64, pub request_count: u64, pub avg_loss_improvement: f64, pub last_access: Instant, pub created_at: Instant, pub last_snapshot_hash: Option<String>, pub decay_lambda: f64, pub effective_ranks: HashMap<String, usize>, pub rank_oracle: Option<RankOracle>, pub adaptation_state: DeltaAdaptationState, /* private fields */
}
Expand description

Per-tenant LoRA delta with per-layer accumulation tracking

Each tenant gets isolated per-layer A/B weight matrices. Keys use the format "layer_idx.module_name" (e.g., "0.q_proj", "27.v_proj"). The delta accumulates across requests and can be periodically saved or merged into a permanent adapter.

Fields§

§lora_a: HashMap<String, Tensor>

LoRA A matrices: “layer_idx.module_name” -> [rank, in_features]

§lora_b: HashMap<String, Tensor>

LoRA B matrices: “layer_idx.module_name” -> [out_features, rank]

§vs: VarStore

VarStore owning all trainable parameters

§muon_states: HashMap<String, MuonState>

Per-parameter Muon optimizer state: VarStore path -> MuonState

§muon_config: MuonConfig

Muon optimizer configuration

§scaling_map: HashMap<String, f64>

Per-key scaling: “layer_idx.module_name” -> alpha / layer_rank Replaces the single scaling field for non-uniform rank support (C7 fix).

§scaling: f64

Default scaling factor: alpha / rank (for uniform-rank deltas or backward compat)

§rank: usize

LoRA rank

§device: Device

Device for tensors

§target_modules: Vec<String>

Target module names (without layer prefix)

§num_layers: usize

Number of model layers

§learning_rate: f64

Learning rate for optimizer

§accumulated_steps: u64

Total gradient steps accumulated

§max_accumulated_steps: u64

Maximum steps before requiring save/merge

§request_count: u64

Number of inference requests served

§avg_loss_improvement: f64

Running average of loss improvement per adaptation

§last_access: Instant

Last time this delta was accessed

§created_at: Instant

When this delta was created

§last_snapshot_hash: Option<String>

Hash of last CAS snapshot (if any)

§decay_lambda: f64

Weight decay factor (used by Muon’s decoupled weight decay)

§effective_ranks: HashMap<String, usize>

Per-key effective rank (for narrow-based rank adaptation). Key = “layer_idx.module_name”, value = active rank (1..=allocated_rank). Defaults to the allocated rank for each key.

§rank_oracle: Option<RankOracle>

Optional per-tenant rank oracle for runtime rank adaptation

§adaptation_state: DeltaAdaptationState

Lifecycle state of any pending adaptation (Idle or Pending with snapshot).

Implementations§

Source§

impl TenantDelta

Source

pub fn new( config: &TenantDeltaConfig, module_dims: &HashMap<String, (usize, usize)>, device: Device, num_layers: usize, ) -> Result<TenantDelta, Error>

Create a new per-layer tenant delta with Kaiming init for A and zeros for B.

Creates num_layers × num_modules A/B pairs keyed as "layer_idx.module_name".

§Arguments
  • config - Delta configuration (rank, alpha, target modules, layer_overrides, etc.)
  • module_dims - Map of module_name -> (in_features, out_features) (flat, for uniform layers)
  • device - Device for tensor allocation
  • num_layers - Number of model layers
  • per_layer_dims - Optional per-layer dim overrides: layer_idx -> module_name -> (in, out) Required when different layer types have different o_proj dimensions (e.g. Qwen3.5).
Source

pub fn new_with_per_layer_dims( config: &TenantDeltaConfig, module_dims: &HashMap<String, (usize, usize)>, device: Device, num_layers: usize, per_layer_dims: Option<&HashMap<usize, HashMap<String, (usize, usize)>>>, ) -> Result<TenantDelta, Error>

Create a new per-layer tenant delta with optional per-layer dimension overrides.

Same as new() but accepts per_layer_dims for architectures where o_proj (or other modules) have different dimensions across layer types.

Source

pub fn has_module(&self, module_name: &str, layer_idx: usize) -> bool

Check if this delta has a module at the given layer

Source

pub fn forward( &self, x: &Tensor, module_name: &str, layer_idx: usize, ) -> Result<Tensor, Error>

Compute LoRA correction for a given module at a specific layer: output += scaling * (x @ A^T) @ B^T

§Arguments
  • x - Input tensor [batch, seq_len, in_features]
  • module_name - Name of the target module
  • layer_idx - Layer index
§Returns

LoRA correction tensor [batch, seq_len, out_features]

Source

pub fn forward_2d( &self, x: &Tensor, module_name: &str, layer_idx: usize, ) -> Result<Tensor, Error>

Compute LoRA correction for 2D tensors at a specific layer: output += scaling * (x @ A^T) @ B^T

Handles shape [batch*seq_len, features] used inside attention layers.

Multi-precision: Input x is cast to a.kind() (always Float32) via x.to_kind(). LoRA A/B are kept in Float32 regardless of the base model’s dtype. This ensures Newton-Schulz convergence in the Muon optimizer and gradient precision within TTT’s tight 1-5 step budget. See also muon::ns_compute_kind().

§Arguments
  • x - Input tensor [tokens, in_features]
  • module_name - Name of the target module (e.g., “q_proj”, “v_proj”)
  • layer_idx - Layer index
§Returns

LoRA correction tensor [tokens, out_features]

Source

pub fn delta_norm_ratio( &self, base_norms: &HashMap<String, f64>, ) -> HashMap<String, f64>

Compute the ratio of delta norm to a reference norm for drift monitoring

Returns a map of “layer_idx.module_name” -> ||delta|| / ||base|| where delta is the effective weight change (B @ A) and base is the reference norm.

Source

pub fn extract_state_dict(&self) -> HashMap<String, Tensor>

Extract state dict (A and B tensors, effective ranks, Muon momentum) for serialization

Keys use the format "layer_idx.module_name.lora_a" / "layer_idx.module_name.lora_b". Effective ranks are stored as "layer_idx.module_name.__effective_rank" (scalar i64). Muon momentum buffers are stored as "layer_idx.module_name.__muon_momentum".

Source

pub fn load_state_dict( &mut self, state: &HashMap<String, Tensor>, ) -> Result<(), Error>

Load state dict (restore A/B tensors, effective ranks, and Muon momentum from a snapshot)

Expects keys in format "layer_idx.module_name.lora_a" / "layer_idx.module_name.lora_b". Also restores __effective_rank and __muon_momentum metadata if present.

Source

pub fn touch(&mut self)

Update last access time

Source

pub fn weight_epoch(&self) -> u64

Current weight epoch (version counter for KV-cache staleness detection).

Source

pub fn bump_weight_epoch(&mut self)

Bump the weight epoch. Must be called on every in-place mutation of the effective weights that happens without a remove/re-add of the delta — adaptations, train steps, rollback restores, resets. O(1); the session KV layer compares epochs on reuse and clears weight-stale caches.

Source

pub fn memory_bytes(&self) -> usize

Estimate memory usage in bytes

Source

pub fn is_at_capacity(&self) -> bool

Check if the delta has reached its accumulation limit

Source

pub fn reset(&mut self)

Reset the delta to zeros (re-initialize B matrices, keep A)

Source

pub fn zero_grad(&self)

Zero all gradients on trainable variables.

Source

pub fn effective_rank(&self, key: &str) -> Option<usize>

Get the effective (active) rank for a key. Returns None if key doesn’t exist.

Source

pub fn set_effective_rank(&mut self, key: &str, rank: usize)

Set the effective rank for a key. Clamped to [1, max_rank]. Also updates the scaling factor for this key (alpha / effective_rank).

Source§

impl TenantDelta

Source

pub fn compose( base: &Arc<Mutex<RawMutex, TenantDelta>>, tenant: &Arc<Mutex<RawMutex, TenantDelta>>, ) -> Arc<Mutex<RawMutex, TenantDelta>> ⓘ

Compose two deltas: base + tenant

Creates a new “view” delta whose forward pass sums corrections from both. The composed delta is non-trainable (for inference only).

Both deltas’ A/B matrices are summed element-wise for shared keys. For keys only in one delta, those corrections are used as-is.

Source§

impl TenantDelta

Source

pub fn serialize_to_safetensors_bytes(&self) -> Result<Vec<u8>, Error>

Serialize per-layer LoRA A/B matrices to safetensors bytes (in memory).

Uses standard HuggingFace PEFT naming convention: base_model.model.layers.{N}.{self_attn|mlp}.{module}.lora_{A|B}.weight

Source

pub fn load_from_safetensors_bytes( data: &[u8], device: Device, ) -> Result<TenantDelta, Error>

Load per-layer LoRA from safetensors bytes (in memory).

Parses standard HuggingFace PEFT naming conventions with layer indices.

Trait Implementations§

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<T> AnySync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

§

impl<'a, T, E> AsTaggedExplicit<'a, E> for T
where T: 'a,

§

fn explicit(self, class: Class, tag: u32) -> TaggedParser<'a, Explicit, Self, E>

§

impl<'a, T, E> AsTaggedImplicit<'a, E> for T
where T: 'a,

§

fn implicit( self, class: Class, constructed: bool, tag: u32, ) -> TaggedParser<'a, Implicit, Self, E>

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<T> JaggedSlice<T> for T

§

type Index = Index2

§

fn push_into(self, array: &mut Jagged<T>)

§

fn insert_into(self, index: <T as JaggedSlice<T>>::Index, array: &mut Jagged<T>)

§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> SendSyncUnwindSafe for T
where T: Send + Sync + UnwindSafe + ?Sized,