Skip to main content

Module runtime

Module runtime 

Source
Expand description

Runtime abstraction layer for inference engines

This module provides a unified interface for different inference engines:

  • TorchEngine: Primary PyTorch-based engine with tch-rs
  • CandleEngine: Legacy Candle engine (DEPRECATED)
  • LlamaCppEngine: Legacy engine for reference (DEPRECATED)

Modules§

architectures
Architecture-specific model implementations
batched_lora
Batched multi-tenant LoRA forward pass
device_pool
Multi-GPU device abstraction (DevicePool).
generation_metrics
Generation quality metrics for self-supervised training
image_utils
Image loading and preprocessing utilities for multimodal models
inference_profile
Deployment and scheduling contract for inference-engine instances.
inference_stage
Thread-local execution boundary for one contiguous decoder stage.
kv_cache
Key-Value cache implementation for efficient autoregressive generation
kv_compat
KV-cache compatibility fingerprint and rejection policy.
model_config
Unified model configuration management
model_factory
Model factory for creating models with unified configuration management
rope
Rotary Position Embedding (RoPE) implementation for tch-rs
template_engine
Jinja2 template engine for chat templates using minijinja
tensor_helpers
tensor_sampling
Device-agnostic tensor sampling for token generation
token_metrics
OpenTelemetry token-burn metering for inference.
tokenizer_config
Trait-based tokenizer configuration for different model architectures
torch_engine
PyTorch-based inference engine using tch-rs
torch_utils
Utilities for safe PyTorch operations with proper error handling
ttn_profile
TTN (Test-Time Naturalization) analysis pipeline for model layer profiling.
weight_provider
Weight provider for streaming large models

Structs§

CacheConfig
Configuration for KV cache instances
DevicePool
A resolved, validated set of compute devices a single inference service owns.
Fp8ExecutionError
Recoverable FP8 execution failure exposed by the native model forward path.
GenerationConfig
Text generation parameters
GenerationMetricsAccumulator
O(1) online metrics accumulator using Welford’s algorithm
GenerationQualityMetrics
Aggregated quality metrics from a generation (~24 bytes)
GenerationResult
Generation result with metadata
GenerationStats
Statistics about text generation with separate prefill/inference metrics
InferenceStage
A loaded, same-thread decoder stage.
KVCacheManager
KV cache manager for all layers in a model
KVCacheRegistry
Registry managing multiple KV cache instances for concurrent sessions.
KvCompatDescriptor
The complete set of authoritative inputs that determine whether KV states computed under one configuration can be reused under another.
KvCompatFingerprint
Content-addressed digest (SHA-256) of a KvCompatDescriptor.
MistralEngine
ModelConfig
Model loading and identification
RuntimeConfig
Runtime execution configuration
SessionMetrics
Per-session metrics storage for tracking quality across conversation turns
StageContract
The range and endpoint responsibilities of one loaded decoder stage.
StageSequence
The single cache timeline owned by one loaded stage.
StageTensor
A stage result that remains attached to the engine-owning thread.
TextStream
Stream that yields decoded UTF-8 text chunks during generation.
TorchEngine
PyTorch inference engine using tch-rs
WeightIdentity
Authoritative identity of the effective weights (base + adapter/delta) that produced a cache’s KV.

Enums§

CacheOwner
Identifies the owner of a KV cache instance for session isolation.
FinishReason
Why generation stopped
KVQuantType
Generated from Cap’n Proto enum KVQuantType
KvCompatMismatch
Why a cache was rejected for reuse, naming the first differing authoritative field in a fixed comparison order. Produced by check_compatibility.
KvDtype
Compute dtype the model stores K/V in.
KvQuantMode
KV-cache storage encoding / quantization mode.
ModelBuilderConfig
StageInput
Input admitted at a stage boundary.
StageOutput
Result emitted at a stage boundary.

Constants§

KV_COMPAT_FORMAT_VERSION
Wire/layout format version of the compatibility fingerprint.

Traits§

RuntimeEngine
Core runtime engine trait - all engines implement this

Functions§

check_compatibility
Rejection policy: a cache whose descriptor differs in any authoritative field is rejected.
create_engine
Create the default runtime engine (now uses PyTorch)