Skip to main content

Module delta_reducer

Module delta_reducer 

Source
Expand description

N-way delta aggregation (param-server reducer) — replaces DO-Merge for multi-contributor merges.

§Why this exists

crate::training::merge’s DO-Merge (do_merge) and additive_merge assert shape equality on the raw LoRA A/B factors. The moment two contributors carry different per-layer ranks — the normal case once the rank oracle (ttn_profile) assigns non-uniform per-layer ranks, or once different workers run with different layer_overrides — those paths hard-fail. They are also non-associative (path-dependent blend weights + direction renormalization), so an N-way merge would depend on contributor order.

This module replaces the N-way aggregation with an order-independent, rank-heterogeneity-tolerant reducer. The 1-vs-1 save-time DO-Merge path may stay; this is the path used when several independent TTT contributors are aggregated each round.

§Algorithm (stack + truncated SVD in delta-space)

For one layer key, each contributor i supplies factors A_i [r_i, in], B_i [out, r_i] and a fold-in weight w_i (provenance trust × norm-bound × per-key scaling). The aggregate effective weight is

  ΔW = Σ_i w_i · (B_i · A_i)
     = B_stack · A_stack

where B_stack = [ w_1·B_1 | w_2·B_2 | … ] is [out, Σr_i] and A_stack = [ A_1 ; A_2 ; … ] is [Σr_i, in]. This is exact: stacking the factors is the weighted sum of the rank-1 updates, with no dense [out, in] materialization and no requirement that the r_i agree.

We then take one rank-R truncated SVD of ΔW per round, computed on the tall-skinny stacked factors (Σr is the sum of contributor ranks — small) without ever forming the dense [out, in]:

  1. economy SVD of A_stack [Σr, in] → A_stack = Uₐ·diag(Sₐ)·Vhₐ (factors are [Σr, Σr], [Σr], [Σr, in] since Σr ≪ in);
  2. C = B_stack · Uₐ · diag(Sₐ) → [out, Σr] (small, no dense out×in);
  3. economy SVD of C [out, Σr] → C = U_c·diag(S_c)·Vh_c;
  4. ΔW = U_c · diag(S_c) · (Vh_c · Vhₐ) — an exact SVD of ΔW;
  5. truncate to R: B_out = U_c[:, :R]·diag(S_c[:R]) [out, R], A_out = (Vh_c·Vhₐ)[:R, :] [R, in].

The reduced factors are stored with scaling = 1.0 (the weights and per-contributor scaling are already folded in). Output rank is the fixed, operational R — stable checkpoint shapes.

§Order-independence

ΔW is a plain sum, so permuting contributors permutes columns of B_stack and rows of A_stack identically and leaves the product ΔW unchanged; the SVD of ΔW is therefore identical up to fp rounding.

§Provenance / norm gate (secure multitenant)

Aggregation is over an explicit contributor set — the reducer never discovers or mixes deltas across tenants. Each contributor carries a verified host id (Phase-2 mesh trust, #354) and a delta_norm_ratio. The ContributionGate rejects (does not silently drop or clamp-to-zero) any contributor whose norm exceeds the bound, and weights survivors by trust × f(norm_ratio). A pluggable ValidationHook seam is provided for the deferred held-out-eval host (default impl is a no-op).

§Send/Sync

tch Tensor is !Send. The reducer keeps all tensor work on a single device/thread (reduce() takes &[Contribution] by reference and returns freshly-built tensors; it never moves a Tensor across a thread boundary). Contributions cross thread/host boundaries as serialized delta bytes (safetensors) + plain-old-data provenance, deserialized on the reducer thread — never as live Tensors.

§Optimizer state

Per the spike, Muon momentum is dropped at aggregation: averaging optimizer state across independent TTT runs is unsound. The reduced delta starts with a fresh (empty) optimizer.

Structs§

Contribution
One contributor in an N-way aggregation round.
ContributionGate
Gate + weighting policy applied to each contributor before it enters the stack. Anomalous / over-norm contributors are rejected (error), never silently dropped or clamped to zero — a quiet drop would let a poisoned host shrink the merge invisibly.
ContributorState
One contributor’s full state for reduce_state_dicts.
NoopValidation
No-op validation hook (the default until the held-out-eval host lands).
Provenance
Provenance for one contributor’s delta.
StackSvdReducer
Stack-in-delta-space + one truncated SVD per round to a fixed rank R.

Traits§

DeltaReducer
Order-independent N-way delta reducer.
ValidationHook
Pluggable post-reduce validation seam.

Functions§

rank_cap_for_dtype_bytes
Static rank↔quant coupling cap.
reduce_state_dicts
Aggregate N contributors’ full state dicts into one reduced state dict.