Expand description
Pipeline-split (2b) training orchestration for TTT-on-split (#316).
This is the training counterpart to the inference forward_layers pipeline
(#314): it drives a TTT/training forward across a set of per-device stage
models, building a single autograd graph over the
crate::runtime::device_pool::LayerDeviceMap so that loss.backward()
traverses the same partition inference uses (M-TRAIN-COUPLING). The only
cross-device copies are the stage-boundary hidden.to_device(next) transfers
inside each stage’s forward_layers_train, which tch makes
autograd-transparent — gradients flow back across device boundaries to each
parameter’s own device.
§Stage contract (mirrors the inference pipeline)
Stages are contiguous global layer ranges [a..b) derived from the device
map. The first stage owns embed_tokens, the last stage owns the final norm
and LM head:
- stage 0 :
embed_tokens→forward_layers_train(0..b) - middle :
forward_layers_train(a..b) - last :
forward_layers_train(a..N)→apply_final_norm→lm_head
Each stage’s training runner uses no KV cache, full causal attention,
start_pos = 0, and fresh call-local recurrent (SSM) state — see
crate::runtime::architectures::ModelOperations::forward_layers_train.
§Send / !Send
Like the inference pipeline, the stage models hold !Send tch tensors and
must all live on the one engine-owning thread; this module never crosses a
thread boundary. Multi-device-on-one-thread is fine (libtorch DeviceGuard).
The single-stage case (stages.len() == 1 over a single-device map) is the
unsplit fast path and is byte-identical to whole-model training.
Structs§
- Train
Stage - A single pipeline stage: the stage model plus the global layer range it owns.
Functions§
- compute_
ntp_ loss_ split - Compute the next-token-prediction (NTP) cross-entropy loss for a TTT step over a pipeline split, with the autograd graph intact across device boundaries.
- forward_
train_ pipeline - Run the TTT/training forward across pipeline stages and return logits with the autograd graph intact across device boundaries.
- stage_
ranges - Derive contiguous stage ranges from a
LayerDeviceMap.