Expand description
Runtime abstraction layer for inference engines
This module provides a unified interface for different inference engines:
- TorchEngine: Primary PyTorch-based engine with tch-rs
- CandleEngine: Legacy Candle engine (DEPRECATED)
- LlamaCppEngine: Legacy engine for reference (DEPRECATED)
Modules§
- architectures
- Architecture-specific model implementations
- batched_
lora - Batched multi-tenant LoRA forward pass
- device_
pool - Multi-GPU device abstraction (
DevicePool). - generation_
metrics - Generation quality metrics for self-supervised training
- image_
utils - Image loading and preprocessing utilities for multimodal models
- inference_
profile - Deployment and scheduling contract for inference-engine instances.
- inference_
stage - Thread-local execution boundary for one contiguous decoder stage.
- kv_
cache - Key-Value cache implementation for efficient autoregressive generation
- kv_
compat - KV-cache compatibility fingerprint and rejection policy.
- model_
config - Unified model configuration management
- model_
factory - Model factory for creating models with unified configuration management
- rope
- Rotary Position Embedding (RoPE) implementation for tch-rs
- template_
engine - Jinja2 template engine for chat templates using minijinja
- tensor_
helpers - tensor_
sampling - Device-agnostic tensor sampling for token generation
- token_
metrics - OpenTelemetry token-burn metering for inference.
- tokenizer_
config - Trait-based tokenizer configuration for different model architectures
- torch_
engine - PyTorch-based inference engine using tch-rs
- torch_
utils - Utilities for safe PyTorch operations with proper error handling
- ttn_
profile - TTN (Test-Time Naturalization) analysis pipeline for model layer profiling.
- weight_
provider - Weight provider for streaming large models
Structs§
- Cache
Config - Configuration for KV cache instances
- Device
Pool - A resolved, validated set of compute devices a single inference service owns.
- Fp8Execution
Error - Recoverable FP8 execution failure exposed by the native model forward path.
- Generation
Config - Text generation parameters
- Generation
Metrics Accumulator - O(1) online metrics accumulator using Welford’s algorithm
- Generation
Quality Metrics - Aggregated quality metrics from a generation (~24 bytes)
- Generation
Result - Generation result with metadata
- Generation
Stats - Statistics about text generation with separate prefill/inference metrics
- Inference
Stage - A loaded, same-thread decoder stage.
- KVCache
Manager - KV cache manager for all layers in a model
- KVCache
Registry - Registry managing multiple KV cache instances for concurrent sessions.
- KvCompat
Descriptor - The complete set of authoritative inputs that determine whether KV states computed under one configuration can be reused under another.
- KvCompat
Fingerprint - Content-addressed digest (SHA-256) of a
KvCompatDescriptor. - Mistral
Engine - Model
Config - Model loading and identification
- Runtime
Config - Runtime execution configuration
- Session
Metrics - Per-session metrics storage for tracking quality across conversation turns
- Stage
Contract - The range and endpoint responsibilities of one loaded decoder stage.
- Stage
Sequence - The single cache timeline owned by one loaded stage.
- Stage
Tensor - A stage result that remains attached to the engine-owning thread.
- Text
Stream - Stream that yields decoded UTF-8 text chunks during generation.
- Torch
Engine - PyTorch inference engine using tch-rs
- Weight
Identity - Authoritative identity of the effective weights (base + adapter/delta) that produced a cache’s KV.
Enums§
- Cache
Owner - Identifies the owner of a KV cache instance for session isolation.
- Finish
Reason - Why generation stopped
- KVQuant
Type - Generated from Cap’n Proto enum KVQuantType
- KvCompat
Mismatch - Why a cache was rejected for reuse, naming the first differing authoritative
field in a fixed comparison order. Produced by
check_compatibility. - KvDtype
- Compute dtype the model stores K/V in.
- KvQuant
Mode - KV-cache storage encoding / quantization mode.
- Model
Builder Config - Stage
Input - Input admitted at a stage boundary.
- Stage
Output - Result emitted at a stage boundary.
Constants§
- KV_
COMPAT_ FORMAT_ VERSION - Wire/layout format version of the compatibility fingerprint.
Traits§
- Runtime
Engine - Core runtime engine trait - all engines implement this
Functions§
- check_
compatibility - Rejection policy: a cache whose descriptor differs in any authoritative field is rejected.
- create_
engine - Create the default runtime engine (now uses PyTorch)