pub struct RuntimeConfig {Show 22 fields
pub context_length: usize,
pub max_context: Option<u32>,
pub kv_quant_type: KVQuantType,
pub batch_size: usize,
pub cpu_threads: Option<usize>,
pub use_gpu: bool,
pub gpu_device_id: Option<usize>,
pub devices: Vec<usize>,
pub strict_device: bool,
pub gpu_layers: Option<usize>,
pub mmap: bool,
pub kv_cache_size_mb: usize,
pub precision_mode: Option<String>,
pub max_concurrent_loads: usize,
pub max_concurrent_generations: usize,
pub default_generation_timeout_ms: u64,
pub default_model_load_timeout_ms: u64,
pub continuous_batching: bool,
pub continuous_batch_max: usize,
pub fp8_gemm: bool,
pub speculative_decoding: bool,
pub fp8_dequant_load: bool,
}Expand description
Runtime execution configuration
Fields§
§context_length: usizeContext window size
max_context: Option<u32>Maximum context length override for KV cache allocation. None = use model’s max_position_embeddings (can be very large, e.g., 40K tokens) Some(n) = cap KV cache at n tokens (significantly reduces GPU memory)
kv_quant_type: KVQuantTypeKV cache quantization type (None, INT8, NF4, FP4). Reduces GPU memory by 50-75% at slight quality cost.
batch_size: usizeBatch processing size
cpu_threads: Option<usize>CPU threads (None = auto-detect)
use_gpu: boolUse GPU acceleration
gpu_device_id: Option<usize>GPU device ID (None = auto-detect, typically device 0).
Legacy single-GPU selector. For multi-GPU, prefer Self::devices;
this field remains the back-compat fallback when devices is empty.
devices: Vec<usize>Explicit set of GPU device indices for multi-GPU (#313, epic #310).
Empty = unset → fall back to the single Self::gpu_device_id /
HYPRSTREAM_GPU_DEVICE (existing single-GPU behavior is unchanged).
Parsed from HYPRSTREAM_GPU_DEVICES (comma-separated, e.g. 0,1).
Resolution + validation lives in Self::resolve_device_indices and is
consumed by runtime::DevicePool.
strict_device: boolFail fast when a requested GPU is unavailable instead of silently downgrading to CPU (#315, epic #310).
A process told to run on GPU 3 that silently lands on CPU tanks a pipeline
split, so strictness is the safe default for the multi-GPU path. This only
affects the case where a GPU was explicitly requested (use_gpu with an
explicit gpu_device_id/devices); pure auto-detect (use_gpu with no
device requested) still falls back to CPU so the legacy single-GPU
“use a GPU if there is one” behavior is unchanged.
Defaults to true; override with HYPRSTREAM_STRICT_DEVICE=0.
gpu_layers: Option<usize>GPU layers to offload (None = auto)
mmap: boolUse memory mapping for model files
kv_cache_size_mb: usizeKV cache size in MB
precision_mode: Option<String>Case-insensitive compute policy: auto, BF16/bfloat16, FP16/float16, FP32/float32, or FP8. FP8 describes weight-path selection: compute stays BF16 except channel-FP8 CPU activation-QDQ/matmuls, which use Float32. Checkpoint quantization remains independently validated from its tensors.
max_concurrent_loads: usize§max_concurrent_generations: usize§default_generation_timeout_ms: u64§default_model_load_timeout_ms: u64§continuous_batching: boolContinuous / in-flight batching (#329, epic #310). Default: off.
When enabled, the inference scheduler groups concurrent decode steps of
same-tenant-delta sequences into a single batched forward (Llama only).
When off, each stream runs the unchanged batch=1 decode path. Override
with HYPRSTREAM_CONTINUOUS_BATCH (truthy = on). Off by default while the
scheduler wiring lands incrementally — the batched kernel is correctness-
gated by batched_ragged_decode_matches_serial.
continuous_batch_max: usizeMax sequences fused into one batched decode step when
Self::continuous_batching is on (spike default 16). Tunable via
HYPRSTREAM_CONTINUOUS_BATCH_MAX.
fp8_gemm: boolFP8 GEMM via torch _scaled_mm for FP8 (e4m3) weight projections.
Default: off.
When enabled, LinearProjection::apply computes FP8-weight matmuls
with at::_scaled_mm instead of the lazy BF16 dequant-then-matmul,
picking the first recipe the device supports (torch 2.11, verified
against release/2.11 ScaledBlas.cpp / RowwiseScaledMM.cu):
- v2 blockwise (1x128 activation × 128x128 weight blocks, the
checkpoint’s native scales): NVIDIA Hopper (SM90) + cuBLASLt ≥
12.9 only —
_check_deepseek_supporthard-errors on SM120, SM100, SM89, and there is no CPU kernel. - v1 rowwise (per-token × per-output-channel scales): SM90+,
including SM120/Blackwell — cuBLASLt rowwise at cuBLAS ≥ 12.9,
otherwise the CUTLASS
f8f8bf16_rowwiseSM120 kernel. Requires a load-time requantization of the weight to per-output-channel scales (coarser than 128x128 blocks; +1x FP8 weight memory while the flag is on).
Each recipe latches off per device after its first kernel error and the runtime falls back to lazy dequant, so enabling this on unsupported hardware costs one warning per device per recipe, not a failure.
Precedence note: like mmap, this field is currently env-only in
effect — model construction has no RuntimeConfig plumbing, so the
runtime reads HYPRSTREAM_FP8_GEMM directly (cached per process);
setting the field programmatically does not reach the model path.
speculative_decoding: boolSelf-speculative decoding via the Qwen3.5 MTP draft head. Default: off.
When enabled (and the loaded model is a dense Qwen3.5 checkpoint with an
MTP head), each decode step drafts 1 token with the MTP head and verifies
it in the next main-model forward: a greedy exact-match accept emits 2
tokens, a reject emits the verifier’s token and rewinds KV/SSM state.
v1 restrictions: batch=1 streams only, greedy sampling only
(temperature ≤ 0.01), no tenant LoRA delta, no quantized KV cache
(truncate_to is a full clear for quantized storage, which would silently
break the reject rewind), dense (non-MoE) MTP blocks only. Override with
HYPRSTREAM_SPECULATIVE_DECODE (truthy = on). Correctness is gated by
mtp_decode_matches_serial.
§Performance envelope (measured)
Experimental — measured regression, keep default-off. GPU validation (RTX 5090 / SM120, Qwen3.5-4B BF16) found the path functionally correct but 0.237x serial throughput (48.84% draft acceptance). The naive cost model — accept = 1 forward per 2 tokens, reject = 2 forwards per 1 token, breakeven at α ≈ 0.5 acceptance with speedup ≈ (1+α)/(2−α) — holds only BEFORE structural per-round overheads:
- a full GDN conv/rec SSM-state deep copy (
snapshot_ssm_states) every round — fixed cost per round, suspected dominant (profile this first if this is ever optimized); - the reject-path single-token re-forward that re-syncs SSM state;
- the per-round MTP draft forward itself.
At 4B with k=1 these overheads dominate the verify savings; break-even
needs the verification cost amortized (larger backbone) or a cheaper
draft path. Enable only per workload after measuring acceptance via the
inference_speculative_tokens_total counter (kind=accepted|rejected);
if a workload shows <~60–70% acceptance this flag is a pessimization.
fp8_dequant_load: boolMaterialize FP8 weights as BF16 once at load time (applying the
block-wise _scale_inv scales during load) and drop the FP8+scale
tensors, instead of dequantizing inside every matmul on the hot path.
Trades ~2x weight VRAM for zero per-matmul dequant work. Tunable via
HYPRSTREAM_FP8_DEQUANT_LOAD (truthy = on). Off by default — off keeps
the FP8-in-VRAM behavior, which is required when the BF16 equivalent
would exceed VRAM. Ignored (with a warning) for multi-device pipelines:
weights are loaded onto the pool’s primary device before layers are
distributed, so full-model BF16 materialization there could OOM.
Implementations§
Source§impl RuntimeConfig
impl RuntimeConfig
Sourcepub fn resolve_explicit_multi_device_indices(
&self,
) -> Result<Option<Vec<usize>>>
pub fn resolve_explicit_multi_device_indices( &self, ) -> Result<Option<Vec<usize>>>
Resolve the explicitly-requested multi-GPU device set, fail-fast.
This is the seam the multi-GPU foundation uses to decide whether to engage
the new DevicePool path. It considers only the explicit multi-GPU
inputs and deliberately excludes the legacy single Self::gpu_device_id
so that existing single-GPU behavior is left entirely on its old code
path (#313 introduces the pool without changing single-GPU runtime
behavior). Precedence:
HYPRSTREAM_GPU_DEVICESenv var (re-parsed here strictly, so a malformed value is a hard error rather than a silent fallback).- The
Self::devicesfield (e.g. from a config file / CLI).
Returns Ok(None) when no explicit multi-GPU set was requested,
Ok(Some(indices)) (non-empty, duplicate-free) otherwise, and Err on
parse errors or duplicate indices.
Sourcepub fn resolve_device_indices(&self) -> Result<Option<Vec<usize>>>
pub fn resolve_device_indices(&self) -> Result<Option<Vec<usize>>>
Resolve the GPU device indices to use, fail-fast, including the legacy single-GPU fallback.
This is the full-precedence resolver consumed by
runtime::DevicePool::from_config. It is
Self::resolve_explicit_multi_device_indices plus a final fallback to
the legacy single Self::gpu_device_id (mapped to a one-element set),
preserving back-compat for callers that build a pool directly from config.
Ok(None) means nothing was requested (caller should auto-detect).
Trait Implementations§
Source§impl Clone for RuntimeConfig
impl Clone for RuntimeConfig
Source§fn clone(&self) -> RuntimeConfig
fn clone(&self) -> RuntimeConfig
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreSource§impl Debug for RuntimeConfig
impl Debug for RuntimeConfig
Source§impl Default for RuntimeConfig
impl Default for RuntimeConfig
Source§impl<'de> Deserialize<'de> for RuntimeConfig
impl<'de> Deserialize<'de> for RuntimeConfig
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Auto Trait Implementations§
impl Freeze for RuntimeConfig
impl RefUnwindSafe for RuntimeConfig
impl Send for RuntimeConfig
impl Sync for RuntimeConfig
impl Unpin for RuntimeConfig
impl UnsafeUnpin for RuntimeConfig
impl UnwindSafe for RuntimeConfig
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more