pub struct LayerDeviceMap { /* private fields */ }Expand description
A per-layer device assignment for 2b intra-host pipeline (#314).
This is the layer→device device_map the DevicePool doc-comment
anticipates. It is a thin, validated newtype over the per-global-layer
Vec<Device> derived from a DevicePool: entry g is the device that
owns global decoder layer g. A stage is a contiguous run of layers that
map to the same device; the only cross-device copy in the forward pass is a
single hidden.to_device(next) at a boundary where consecutive layers map to
different devices (see LayerDeviceMap::is_boundary).
§Why a map over num_hidden_layers, not the stage-local count
The map is global — indexed by global layer index, length
num_hidden_layers — so a single source of truth drives both shard-aware
construction (per-layer placement + per-layer type/RoPE selection, which need
the global index) and the forward loop. A stage that owns layers [a..b)
queries the same global map; the architecture is responsible for the
global↔local remap of its own self.layers / KV / SSM vectors via the
layer_offset = a it was built with.
§Send / Sync
Holds only [Device] values (Copy, no tensors), so it is Send + Sync and
may be handed to an engine-owning thread. Like DevicePool, never add a
Tensor/VarStore field — see the module-level !Send boundary note.
Implementations§
Source§impl LayerDeviceMap
impl LayerDeviceMap
Sourcepub fn single(
device: Device,
num_layers: usize,
) -> Result<LayerDeviceMap, Error>
pub fn single( device: Device, num_layers: usize, ) -> Result<LayerDeviceMap, Error>
All layers on a single device — the unsplit fast path.
Used both by single-GPU inference and by the CPU-only equivalence tests
(forward_layers(0..N) over an all-CPU map must equal the whole-model
forward). num_layers must be non-zero.
Sourcepub fn from_per_layer(per_layer: Vec<Device>) -> Result<LayerDeviceMap, Error>
pub fn from_per_layer(per_layer: Vec<Device>) -> Result<LayerDeviceMap, Error>
Build an explicit per-layer map, validating it is non-empty.
per_layer[g] is the device that owns global layer g.
Sourcepub fn even_split(
pool: &DevicePool,
num_layers: usize,
) -> Result<LayerDeviceMap, Error>
pub fn even_split( pool: &DevicePool, num_layers: usize, ) -> Result<LayerDeviceMap, Error>
Spread num_layers contiguously across a DevicePool’s devices in
requested order, balanced as evenly as possible (capacity-symmetric).
Earlier stages get the extra layer when num_layers is not divisible by
the device count, matching the natural prefix-heavy split. This is the
parameter-balanced planner; the capacity-aware (asymmetric-VRAM) variant
is a later epic concern (plan §H) and would replace this constructor
without touching the forward path.
Layers are assigned in contiguous runs so each device owns a single
pipeline stage [a..b) — never interleaved — which keeps boundary copies
to exactly (num_stages - 1) per forward.
Sourcepub fn device_for(&self, global_layer_idx: usize) -> Device
pub fn device_for(&self, global_layer_idx: usize) -> Device
Device owning global layer global_layer_idx.
§Panics
Panics if global_layer_idx >= len() — a programming error (the caller
iterates a known global range). Use Self::try_device_for for a checked
variant.
Sourcepub fn try_device_for(&self, global_layer_idx: usize) -> Result<Device, Error>
pub fn try_device_for(&self, global_layer_idx: usize) -> Result<Device, Error>
Checked Self::device_for.
Sourcepub fn is_boundary(&self, prev_device: Device, global_layer_idx: usize) -> bool
pub fn is_boundary(&self, prev_device: Device, global_layer_idx: usize) -> bool
Whether a cross-device boundary copy is needed before running layer
global_layer_idx, given the device the previous layer’s output is on.
Returns false for the first layer of a stage when its device equals
prev_device (zero-copy within a stage and across same-device stages), and
true only when the device actually changes. This is the single point
that gates the lone hidden.to_device() in the forward loop.
Sourcepub fn len(&self) -> usize
pub fn len(&self) -> usize
Number of global layers covered (equals the model’s num_hidden_layers).
Sourcepub fn is_single_device(&self) -> bool
pub fn is_single_device(&self) -> bool
Whether every layer maps to the same device (the unsplit fast path).
When true, forward_layers performs zero cross-device copies and is
numerically identical to the single-device whole-model forward.
Trait Implementations§
Source§impl Clone for LayerDeviceMap
impl Clone for LayerDeviceMap
Source§fn clone(&self) -> LayerDeviceMap
fn clone(&self) -> LayerDeviceMap
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreAuto Trait Implementations§
impl Freeze for LayerDeviceMap
impl RefUnwindSafe for LayerDeviceMap
impl Send for LayerDeviceMap
impl Sync for LayerDeviceMap
impl Unpin for LayerDeviceMap
impl UnsafeUnpin for LayerDeviceMap
impl UnwindSafe for LayerDeviceMap
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request