Skip to main content

RuntimeConfig

Struct RuntimeConfig 

Source
pub struct RuntimeConfig {
Show 22 fields pub context_length: usize, pub max_context: Option<u32>, pub kv_quant_type: KVQuantType, pub batch_size: usize, pub cpu_threads: Option<usize>, pub use_gpu: bool, pub gpu_device_id: Option<usize>, pub devices: Vec<usize>, pub strict_device: bool, pub gpu_layers: Option<usize>, pub mmap: bool, pub kv_cache_size_mb: usize, pub precision_mode: Option<String>, pub max_concurrent_loads: usize, pub max_concurrent_generations: usize, pub default_generation_timeout_ms: u64, pub default_model_load_timeout_ms: u64, pub continuous_batching: bool, pub continuous_batch_max: usize, pub fp8_gemm: bool, pub speculative_decoding: bool, pub fp8_dequant_load: bool,
}
Expand description

Runtime execution configuration

Fields§

§context_length: usize

Context window size

§max_context: Option<u32>

Maximum context length override for KV cache allocation. None = use model’s max_position_embeddings (can be very large, e.g., 40K tokens) Some(n) = cap KV cache at n tokens (significantly reduces GPU memory)

§kv_quant_type: KVQuantType

KV cache quantization type (None, INT8, NF4, FP4). Reduces GPU memory by 50-75% at slight quality cost.

§batch_size: usize

Batch processing size

§cpu_threads: Option<usize>

CPU threads (None = auto-detect)

§use_gpu: bool

Use GPU acceleration

§gpu_device_id: Option<usize>

GPU device ID (None = auto-detect, typically device 0).

Legacy single-GPU selector. For multi-GPU, prefer Self::devices; this field remains the back-compat fallback when devices is empty.

§devices: Vec<usize>

Explicit set of GPU device indices for multi-GPU (#313, epic #310).

Empty = unset → fall back to the single Self::gpu_device_id / HYPRSTREAM_GPU_DEVICE (existing single-GPU behavior is unchanged). Parsed from HYPRSTREAM_GPU_DEVICES (comma-separated, e.g. 0,1). Resolution + validation lives in Self::resolve_device_indices and is consumed by runtime::DevicePool.

§strict_device: bool

Fail fast when a requested GPU is unavailable instead of silently downgrading to CPU (#315, epic #310).

A process told to run on GPU 3 that silently lands on CPU tanks a pipeline split, so strictness is the safe default for the multi-GPU path. This only affects the case where a GPU was explicitly requested (use_gpu with an explicit gpu_device_id/devices); pure auto-detect (use_gpu with no device requested) still falls back to CPU so the legacy single-GPU “use a GPU if there is one” behavior is unchanged. Defaults to true; override with HYPRSTREAM_STRICT_DEVICE=0.

§gpu_layers: Option<usize>

GPU layers to offload (None = auto)

§mmap: bool

Use memory mapping for model files

§kv_cache_size_mb: usize

KV cache size in MB

§precision_mode: Option<String>

Case-insensitive compute policy: auto, BF16/bfloat16, FP16/float16, FP32/float32, or FP8. FP8 describes weight-path selection: compute stays BF16 except channel-FP8 CPU activation-QDQ/matmuls, which use Float32. Checkpoint quantization remains independently validated from its tensors.

§max_concurrent_loads: usize§max_concurrent_generations: usize§default_generation_timeout_ms: u64§default_model_load_timeout_ms: u64§continuous_batching: bool

Continuous / in-flight batching (#329, epic #310). Default: off.

When enabled, the inference scheduler groups concurrent decode steps of same-tenant-delta sequences into a single batched forward (Llama only). When off, each stream runs the unchanged batch=1 decode path. Override with HYPRSTREAM_CONTINUOUS_BATCH (truthy = on). Off by default while the scheduler wiring lands incrementally — the batched kernel is correctness- gated by batched_ragged_decode_matches_serial.

§continuous_batch_max: usize

Max sequences fused into one batched decode step when Self::continuous_batching is on (spike default 16). Tunable via HYPRSTREAM_CONTINUOUS_BATCH_MAX.

§fp8_gemm: bool

FP8 GEMM via torch _scaled_mm for FP8 (e4m3) weight projections. Default: off.

When enabled, LinearProjection::apply computes FP8-weight matmuls with at::_scaled_mm instead of the lazy BF16 dequant-then-matmul, picking the first recipe the device supports (torch 2.11, verified against release/2.11 ScaledBlas.cpp / RowwiseScaledMM.cu):

  1. v2 blockwise (1x128 activation × 128x128 weight blocks, the checkpoint’s native scales): NVIDIA Hopper (SM90) + cuBLASLt ≥ 12.9 only — _check_deepseek_support hard-errors on SM120, SM100, SM89, and there is no CPU kernel.
  2. v1 rowwise (per-token × per-output-channel scales): SM90+, including SM120/Blackwell — cuBLASLt rowwise at cuBLAS ≥ 12.9, otherwise the CUTLASS f8f8bf16_rowwise SM120 kernel. Requires a load-time requantization of the weight to per-output-channel scales (coarser than 128x128 blocks; +1x FP8 weight memory while the flag is on).

Each recipe latches off per device after its first kernel error and the runtime falls back to lazy dequant, so enabling this on unsupported hardware costs one warning per device per recipe, not a failure.

Precedence note: like mmap, this field is currently env-only in effect — model construction has no RuntimeConfig plumbing, so the runtime reads HYPRSTREAM_FP8_GEMM directly (cached per process); setting the field programmatically does not reach the model path.

§speculative_decoding: bool

Self-speculative decoding via the Qwen3.5 MTP draft head. Default: off.

When enabled (and the loaded model is a dense Qwen3.5 checkpoint with an MTP head), each decode step drafts 1 token with the MTP head and verifies it in the next main-model forward: a greedy exact-match accept emits 2 tokens, a reject emits the verifier’s token and rewinds KV/SSM state. v1 restrictions: batch=1 streams only, greedy sampling only (temperature ≤ 0.01), no tenant LoRA delta, no quantized KV cache (truncate_to is a full clear for quantized storage, which would silently break the reject rewind), dense (non-MoE) MTP blocks only. Override with HYPRSTREAM_SPECULATIVE_DECODE (truthy = on). Correctness is gated by mtp_decode_matches_serial.

§Performance envelope (measured)

Experimental — measured regression, keep default-off. GPU validation (RTX 5090 / SM120, Qwen3.5-4B BF16) found the path functionally correct but 0.237x serial throughput (48.84% draft acceptance). The naive cost model — accept = 1 forward per 2 tokens, reject = 2 forwards per 1 token, breakeven at α ≈ 0.5 acceptance with speedup ≈ (1+α)/(2−α) — holds only BEFORE structural per-round overheads:

  • a full GDN conv/rec SSM-state deep copy (snapshot_ssm_states) every round — fixed cost per round, suspected dominant (profile this first if this is ever optimized);
  • the reject-path single-token re-forward that re-syncs SSM state;
  • the per-round MTP draft forward itself.

At 4B with k=1 these overheads dominate the verify savings; break-even needs the verification cost amortized (larger backbone) or a cheaper draft path. Enable only per workload after measuring acceptance via the inference_speculative_tokens_total counter (kind=accepted|rejected); if a workload shows <~60–70% acceptance this flag is a pessimization.

§fp8_dequant_load: bool

Materialize FP8 weights as BF16 once at load time (applying the block-wise _scale_inv scales during load) and drop the FP8+scale tensors, instead of dequantizing inside every matmul on the hot path. Trades ~2x weight VRAM for zero per-matmul dequant work. Tunable via HYPRSTREAM_FP8_DEQUANT_LOAD (truthy = on). Off by default — off keeps the FP8-in-VRAM behavior, which is required when the BF16 equivalent would exceed VRAM. Ignored (with a warning) for multi-device pipelines: weights are loaded onto the pool’s primary device before layers are distributed, so full-model BF16 materialization there could OOM.

Implementations§

Source§

impl RuntimeConfig

Source

pub fn resolve_explicit_multi_device_indices( &self, ) -> Result<Option<Vec<usize>>, Error>

Resolve the explicitly-requested multi-GPU device set, fail-fast.

This is the seam the multi-GPU foundation uses to decide whether to engage the new DevicePool path. It considers only the explicit multi-GPU inputs and deliberately excludes the legacy single Self::gpu_device_id so that existing single-GPU behavior is left entirely on its old code path (#313 introduces the pool without changing single-GPU runtime behavior). Precedence:

  1. HYPRSTREAM_GPU_DEVICES env var (re-parsed here strictly, so a malformed value is a hard error rather than a silent fallback).
  2. The Self::devices field (e.g. from a config file / CLI).

Returns Ok(None) when no explicit multi-GPU set was requested, Ok(Some(indices)) (non-empty, duplicate-free) otherwise, and Err on parse errors or duplicate indices.

Source

pub fn resolve_device_indices(&self) -> Result<Option<Vec<usize>>, Error>

Resolve the GPU device indices to use, fail-fast, including the legacy single-GPU fallback.

This is the full-precedence resolver consumed by runtime::DevicePool::from_config. It is Self::resolve_explicit_multi_device_indices plus a final fallback to the legacy single Self::gpu_device_id (mapped to a one-element set), preserving back-compat for callers that build a pool directly from config.

Ok(None) means nothing was requested (caller should auto-detect).

Trait Implementations§

Source§

impl Clone for RuntimeConfig

Source§

fn clone(&self) -> RuntimeConfig

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for RuntimeConfig

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter. Read more
Source§

impl Default for RuntimeConfig

Source§

fn default() -> RuntimeConfig

Returns the “default value” for a type. Read more
Source§

impl<'de> Deserialize<'de> for RuntimeConfig

Source§

fn deserialize<__D>( __deserializer: __D, ) -> Result<RuntimeConfig, <__D as Deserializer<'de>>::Error>
where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more
Source§

impl Serialize for RuntimeConfig

Source§

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>
where __S: Serializer,

Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<T> AnySync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

§

impl<'a, T, E> AsTaggedExplicit<'a, E> for T
where T: 'a,

§

fn explicit(self, class: Class, tag: u32) -> TaggedParser<'a, Explicit, Self, E>

§

impl<'a, T, E> AsTaggedImplicit<'a, E> for T
where T: 'a,

§

fn implicit( self, class: Class, constructed: bool, tag: u32, ) -> TaggedParser<'a, Implicit, Self, E>

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> DynClone for T
where T: Clone,

Source§

fn __clone_box(&self, _: Private) -> *mut ()

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FromRef<T> for T
where T: Clone,

§

fn from_ref(input: &T) -> T

Converts to this type from a reference to the input type.
§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<T> JaggedSlice<T> for T

§

type Index = Index2

§

fn push_into(self, array: &mut Jagged<T>)

§

fn insert_into(self, index: <T as JaggedSlice<T>>::Index, array: &mut Jagged<T>)

§

impl<T> NoneValue for T
where T: Default,

§

type NoneType = T

§

fn null_value() -> T

The none-equivalent value.
§

impl<T> NoneValue for T
where T: Default,

§

type NoneType = T

§

fn null_value() -> T

The none-equivalent value.
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> Serialize for T
where T: Serialize + ?Sized,

Source§

fn erased_serialize(&self, serializer: &mut dyn Serializer) -> Result<Ok, Error>

§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
Source§

impl<T> DeserializeOwned for T
where T: for<'de> Deserialize<'de>,

§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<E> ResultError for E
where E: Send + Debug + Sync,

§

impl<T> ResultType for T
where T: Send + Clone + Sync + Debug,

§

impl<T> RpcMessage for T
where T: Debug + Serialize + DeserializeOwned + Send + Sync + Unpin + 'static,

§

impl<T> SendSyncUnwindSafe for T
where T: Send + Sync + UnwindSafe + ?Sized,

Source§

impl<T> TerminalState for T
where T: Clone + Send + Sync,