Skip to main content

Module device_pool

Module device_pool 

Source
Expand description

Multi-GPU device abstraction (DevicePool).

This module introduces the device-strategy seam for the multi-GPU epic (#310, part of #313). It resolves a set of configured device indices (RuntimeConfig.devices) into concrete [tch::Device] values and exposes a small, stable API that the later phases build on:

  • 2a replication (#313): N TorchEngines, one per local GPU, requests routed across DevicePool::devices.
  • 2b intra-host pipeline (#314): a layer→device device_map layered on top of the same pool (see DevicePool note on extensibility).

§Send / !Send boundary

tch-rs tensors are !Send: a Tensor must never cross a thread boundary, and engines that hold tensors stay pinned to one thread (as services/inference.rs already does — one thread per engine). However, a single thread may legitimately own tensors on Cuda(0) and Cuda(1) at once: libtorch dispatches each op to the correct device via a DeviceGuard, so multi-device-on-one-thread is fine; only multi-thread tensor moves are forbidden.

DevicePool deliberately holds only device indices / [tch::Device] values (which are Copy and contain no tensor state). It is therefore Send + Sync and can be shared across threads (e.g. handed to each engine-owning thread so each can construct its own thread-pinned engine). Do not add a Tensor, VarStore, or any !Send field to this type — it would silently make the pool !Send and break the replication design.

Structs§

DevicePool
A resolved, validated set of compute devices a single inference service owns.
LayerDeviceMap
A per-layer device assignment for 2b intra-host pipeline (#314).