Expand description
Multi-GPU device abstraction (DevicePool).
This module introduces the device-strategy seam for the multi-GPU epic (#310,
part of #313). It resolves a set of configured device indices
(RuntimeConfig.devices) into concrete [tch::Device] values and exposes a
small, stable API that the later phases build on:
- 2a replication (#313): N
TorchEngines, one per local GPU, requests routed acrossDevicePool::devices. - 2b intra-host pipeline (#314): a layer→device
device_maplayered on top of the same pool (seeDevicePoolnote on extensibility).
§Send / !Send boundary
tch-rs tensors are !Send: a Tensor must never cross a thread boundary,
and engines that hold tensors stay pinned to one thread (as
services/inference.rs already does — one thread per engine). However, a
single thread may legitimately own tensors on Cuda(0) and Cuda(1)
at once: libtorch dispatches each op to the correct device via a
DeviceGuard, so multi-device-on-one-thread is fine; only multi-thread
tensor moves are forbidden.
DevicePool deliberately holds only device indices / [tch::Device]
values (which are Copy and contain no tensor state). It is therefore
Send + Sync and can be shared across threads (e.g. handed to each
engine-owning thread so each can construct its own thread-pinned engine).
Do not add a Tensor, VarStore, or any !Send field to this type — it
would silently make the pool !Send and break the replication design.
Structs§
- Device
Pool - A resolved, validated set of compute devices a single inference service owns.
- Layer
Device Map - A per-layer device assignment for 2b intra-host pipeline (#314).