pub struct DevicePool { /* private fields */ }Expand description
A resolved, validated set of compute devices a single inference service owns.
Constructed from crate::config::RuntimeConfig via DevicePool::from_config.
The pool performs fail-fast device resolution: when devices are explicitly
requested it never silently downgrades to CPU (“no-fragile-fallbacks”, per the
multi-GPU/multi-host inference design). A process told to use GPU 1 that would
land on CPU is an error, not a warning.
§Send / Sync
DevicePool is Send + Sync because it holds only [Device] values (no
tensors). See the module-level docs for the !Send boundary — keep this type
tensor-free.
§Extensibility (#314)
The pipeline phase needs a layer→device map. That belongs on this type
(e.g. a fn device_for_layer(&self, layer_idx, num_layers) -> Device or an
explicit device_map), reusing the already-resolved Self::devices. New
strategy methods can be added without changing the existing accessors below,
so current callers (which only need Self::primary) keep working.
Implementations§
Source§impl DevicePool
impl DevicePool
Sourcepub fn from_cuda_indices(indices: &[usize]) -> Result<Self>
pub fn from_cuda_indices(indices: &[usize]) -> Result<Self>
Build a pool for the given GPU indices, validating availability fail-fast.
indices must be non-empty and free of duplicates (the caller —
crate::config::RuntimeConfig::resolve_device_indices — guarantees
this, but it is re-checked here so the type’s invariant holds regardless
of construction path).
Every index is validated against the live CUDA/HIP device count. If CUDA is unavailable, or any requested index is out of range, this returns an error rather than degrading to CPU — devices were explicitly requested.
Note: ROCm presents as [Device::Cuda] via HIP, so this path is
vendor-agnostic and contains no vendor-special casing.
Sourcepub fn from_devices(devices: Vec<Device>) -> Result<Self>
pub fn from_devices(devices: Vec<Device>) -> Result<Self>
Construct a pool directly from already-resolved [Device] values.
Intended for tests and for callers that have an explicit device list
(e.g. CPU-only setups). Validates only the structural invariants
(non-empty, no duplicates); it does not probe CUDA, so prefer
Self::from_cuda_indices / Self::from_config for GPU resolution.
Sourcepub fn from_config(config: &RuntimeConfig) -> Result<Self>
pub fn from_config(config: &RuntimeConfig) -> Result<Self>
Resolve a DevicePool from runtime configuration.
Behavior:
use_gpu == false→ a single-entry CPU pool ([Device::Cpu]).use_gpu == true→ resolve the configured indices (crate::config::RuntimeConfig::resolve_device_indices, which honorsdevicesthen falls back to the legacy singlegpu_device_id) and validate them fail-fast viaSelf::from_cuda_indices.
When use_gpu is true but no index is configured, this falls back to
cuda_if_available()-style auto-detection of device 0 (preserving the
pre-existing single-GPU default). That auto-detect path is the only
place a CPU result is allowed when GPU is on, and only because nothing
was explicitly requested.
Sourcepub fn devices(&self) -> &[Device]
pub fn devices(&self) -> &[Device]
All resolved devices, in requested order. Always non-empty.
The primary seam for #313/2a replication (one engine per device).
Sourcepub fn primary(&self) -> Device
pub fn primary(&self) -> Device
The primary device — the first configured device.
Single-device callers (e.g. today’s TorchEngine) use this to keep
existing behavior while the multi-engine routing lands in a later PR.
Sourcepub fn len(&self) -> usize
pub fn len(&self) -> usize
Number of devices in the pool (always >= 1).
No is_empty() companion exists: a DevicePool is never empty by
construction. Use Self::is_single for the meaningful predicate.
Trait Implementations§
Source§impl Clone for DevicePool
impl Clone for DevicePool
Source§fn clone(&self) -> DevicePool
fn clone(&self) -> DevicePool
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreAuto Trait Implementations§
impl Freeze for DevicePool
impl RefUnwindSafe for DevicePool
impl Send for DevicePool
impl Sync for DevicePool
impl Unpin for DevicePool
impl UnsafeUnpin for DevicePool
impl UnwindSafe for DevicePool
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more