Expand description
Runtime abstraction layer for inference engines
This module provides a unified interface for different inference engines:
- TorchEngine: Primary PyTorch-based engine with tch-rs
- CandleEngine: Legacy Candle engine (DEPRECATED)
- LlamaCppEngine: Legacy engine for reference (DEPRECATED)
Re-exports§
pub use crate::config::FinishReason;pub use crate::config::GenerationConfig;pub use crate::config::GenerationResult;pub use crate::config::ModelConfig;pub use crate::config::RuntimeConfig;pub use inference_stage::InferenceStage;pub use inference_stage::StageContract;pub use inference_stage::StageInput;pub use inference_stage::StageOutput;pub use inference_stage::StageSequence;pub use inference_stage::StageTensor;pub use torch_engine::GenerationStats;pub use torch_engine::TextStream;pub use torch_engine::TorchEngine;pub use device_pool::DevicePool;pub use kv_cache::CacheConfig;pub use kv_cache::CacheOwner;pub use kv_cache::KVCacheManager;pub use kv_cache::KVCacheRegistry;pub use kv_compat::check_compatibility;pub use kv_compat::KvCompatDescriptor;pub use kv_compat::KvCompatFingerprint;pub use kv_compat::KvCompatMismatch;pub use kv_compat::KvDtype;pub use kv_compat::KvQuantMode;pub use kv_compat::WeightIdentity;pub use kv_compat::KV_COMPAT_FORMAT_VERSION;pub use generation_metrics::GenerationMetricsAccumulator;pub use generation_metrics::GenerationQualityMetrics;pub use generation_metrics::SessionMetrics;
Modules§
- architectures
- Architecture-specific model implementations
- batched_
lora - Batched multi-tenant LoRA forward pass
- device_
pool - Multi-GPU device abstraction (
DevicePool). - generation_
metrics - Generation quality metrics for self-supervised training
- image_
utils - Image loading and preprocessing utilities for multimodal models
- inference_
profile - Deployment and scheduling contract for inference-engine instances.
- inference_
stage - Thread-local execution boundary for one contiguous decoder stage.
- kv_
cache - Key-Value cache implementation for efficient autoregressive generation
- kv_
compat - KV-cache compatibility fingerprint and rejection policy.
- model_
config - Unified model configuration management
- model_
factory - Model factory for creating models with unified configuration management
- rope
- Rotary Position Embedding (RoPE) implementation for tch-rs
- template_
engine - Jinja2 template engine for chat templates using minijinja
- tensor_
helpers - tensor_
sampling - Device-agnostic tensor sampling for token generation
- token_
metrics - OpenTelemetry token-burn metering for inference.
- tokenizer_
config - Trait-based tokenizer configuration for different model architectures
- torch_
engine - PyTorch-based inference engine using tch-rs
- torch_
utils - Utilities for safe PyTorch operations with proper error handling
- ttn_
profile - TTN (Test-Time Naturalization) analysis pipeline for model layer profiling.
- weight_
provider - Weight provider for streaming large models
Structs§
- Fp8Execution
Error - Recoverable FP8 execution failure exposed by the native model forward path.
- Mistral
Engine
Enums§
- KVQuant
Type - Generated from Cap’n Proto enum KVQuantType
- Model
Builder Config
Traits§
- Runtime
Engine - Core runtime engine trait - all engines implement this
Functions§
- create_
engine - Create the default runtime engine (now uses PyTorch)