Skip to main content

muon_step

Function muon_step 

Source
pub fn muon_step(param: &Tensor, state: &mut MuonState, config: &MuonConfig)
Expand description

Perform a single Muon optimization step on a 2D parameter.

Algorithm:

  1. Nesterov momentum: buf = lerp(buf, grad, 1-momentum); g = lerp(grad, buf, momentum)
  2. Orthogonalize: g = newton_schulz(g) → UV^T (all SVs = 1)
  3. Weight decay: p *= (1 - lr * wd)
  4. Scale for rectangularity: scale = sqrt(max(1, rows/cols))
  5. Update: p -= lr * scale * g