Expand description
N-way delta aggregation (param-server reducer) — replaces DO-Merge for multi-contributor merges.
§Why this exists
crate::training::merge’s DO-Merge (do_merge) and additive_merge
assert shape equality on the raw LoRA A/B factors. The moment two
contributors carry different per-layer ranks — the normal case once the rank
oracle (ttn_profile) assigns non-uniform per-layer ranks, or once
different workers run with different layer_overrides — those paths
hard-fail. They are also non-associative (path-dependent blend weights +
direction renormalization), so an N-way merge would depend on contributor
order.
This module replaces the N-way aggregation with an order-independent, rank-heterogeneity-tolerant reducer. The 1-vs-1 save-time DO-Merge path may stay; this is the path used when several independent TTT contributors are aggregated each round.
§Algorithm (stack + truncated SVD in delta-space)
For one layer key, each contributor i supplies factors A_i [r_i, in],
B_i [out, r_i] and a fold-in weight w_i (provenance trust × norm-bound ×
per-key scaling). The aggregate effective weight is
ΔW = Σ_i w_i · (B_i · A_i)
= B_stack · A_stackwhere B_stack = [ w_1·B_1 | w_2·B_2 | … ] is [out, Σr_i] and
A_stack = [ A_1 ; A_2 ; … ] is [Σr_i, in]. This is exact: stacking the
factors is the weighted sum of the rank-1 updates, with no dense
[out, in] materialization and no requirement that the r_i agree.
We then take one rank-R truncated SVD of ΔW per round, computed on
the tall-skinny stacked factors (Σr is the sum of contributor ranks —
small) without ever forming the dense [out, in]:
- economy SVD of
A_stack [Σr, in]→A_stack = Uₐ·diag(Sₐ)·Vhₐ(factors are[Σr, Σr],[Σr],[Σr, in]sinceΣr ≪ in); C = B_stack · Uₐ · diag(Sₐ)→[out, Σr](small, no denseout×in);- economy SVD of
C [out, Σr]→C = U_c·diag(S_c)·Vh_c; ΔW = U_c · diag(S_c) · (Vh_c · Vhₐ)— an exact SVD ofΔW;- truncate to
R:B_out = U_c[:, :R]·diag(S_c[:R])[out, R],A_out = (Vh_c·Vhₐ)[:R, :][R, in].
The reduced factors are stored with scaling = 1.0 (the weights and
per-contributor scaling are already folded in). Output rank is the fixed,
operational R — stable checkpoint shapes.
§Order-independence
ΔW is a plain sum, so permuting contributors permutes columns of
B_stack and rows of A_stack identically and leaves the product ΔW
unchanged; the SVD of ΔW is therefore identical up to fp rounding.
§Provenance / norm gate (secure multitenant)
Aggregation is over an explicit contributor set — the reducer never
discovers or mixes deltas across tenants. Each contributor carries a
verified host id (Phase-2 mesh trust, #354) and a delta_norm_ratio. The
ContributionGate rejects (does not silently drop or clamp-to-zero)
any contributor whose norm exceeds the bound, and weights survivors by
trust × f(norm_ratio). A pluggable ValidationHook seam is provided for
the deferred held-out-eval host (default impl is a no-op).
§Send/Sync
tch Tensor is !Send. The reducer keeps all tensor work on a single
device/thread (reduce() takes &[Contribution] by reference and returns
freshly-built tensors; it never moves a Tensor across a thread boundary).
Contributions cross thread/host boundaries as serialized delta bytes
(safetensors) + plain-old-data provenance, deserialized on the reducer
thread — never as live Tensors.
§Optimizer state
Per the spike, Muon momentum is dropped at aggregation: averaging optimizer state across independent TTT runs is unsound. The reduced delta starts with a fresh (empty) optimizer.
Structs§
- Contribution
- One contributor in an N-way aggregation round.
- Contribution
Gate - Gate + weighting policy applied to each contributor before it enters the stack. Anomalous / over-norm contributors are rejected (error), never silently dropped or clamped to zero — a quiet drop would let a poisoned host shrink the merge invisibly.
- Contributor
State - One contributor’s full state for
reduce_state_dicts. - Noop
Validation - No-op validation hook (the default until the held-out-eval host lands).
- Provenance
- Provenance for one contributor’s delta.
- Stack
SvdReducer - Stack-in-delta-space + one truncated SVD per round to a fixed rank
R.
Traits§
- Delta
Reducer - Order-independent N-way delta reducer.
- Validation
Hook - Pluggable post-reduce validation seam.
Functions§
- rank_
cap_ for_ dtype_ bytes - Static rank↔quant coupling cap.
- reduce_
state_ dicts - Aggregate N contributors’ full state dicts into one reduced state dict.