Skip to main content

Module pipeline

Module pipeline 

Source
Expand description

Pipeline-split (2b) training orchestration for TTT-on-split (#316).

This is the training counterpart to the inference forward_layers pipeline (#314): it drives a TTT/training forward across a set of per-device stage models, building a single autograd graph over the crate::runtime::device_pool::LayerDeviceMap so that loss.backward() traverses the same partition inference uses (M-TRAIN-COUPLING). The only cross-device copies are the stage-boundary hidden.to_device(next) transfers inside each stage’s forward_layers_train, which tch makes autograd-transparent — gradients flow back across device boundaries to each parameter’s own device.

§Stage contract (mirrors the inference pipeline)

Stages are contiguous global layer ranges [a..b) derived from the device map. The first stage owns embed_tokens, the last stage owns the final norm and LM head:

  • stage 0 : embed_tokens → forward_layers_train(0..b)
  • middle : forward_layers_train(a..b)
  • last : forward_layers_train(a..N) → apply_final_norm → lm_head

Each stage’s training runner uses no KV cache, full causal attention, start_pos = 0, and fresh call-local recurrent (SSM) state — see crate::runtime::architectures::ModelOperations::forward_layers_train.

§Send / !Send

Like the inference pipeline, the stage models hold !Send tch tensors and must all live on the one engine-owning thread; this module never crosses a thread boundary. Multi-device-on-one-thread is fine (libtorch DeviceGuard). The single-stage case (stages.len() == 1 over a single-device map) is the unsplit fast path and is byte-identical to whole-model training.

Structs§

TrainStage
A single pipeline stage: the stage model plus the global layer range it owns.

Functions§

compute_ntp_loss_split
Compute the next-token-prediction (NTP) cross-entropy loss for a TTT step over a pipeline split, with the autograd graph intact across device boundaries.
forward_train_pipeline
Run the TTT/training forward across pipeline stages and return logits with the autograd graph intact across device boundaries.
stage_ranges
Derive contiguous stage ranges from a LayerDeviceMap.