Skip to main content

Module runtime

Module runtime 

Source
Expand description

Runtime abstraction layer for inference engines

This module provides a unified interface for different inference engines:

  • TorchEngine: Primary PyTorch-based engine with tch-rs
  • CandleEngine: Legacy Candle engine (DEPRECATED)
  • LlamaCppEngine: Legacy engine for reference (DEPRECATED)

Re-exports§

pub use crate::config::FinishReason;
pub use crate::config::GenerationConfig;
pub use crate::config::GenerationResult;
pub use crate::config::ModelConfig;
pub use crate::config::RuntimeConfig;
pub use inference_stage::InferenceStage;
pub use inference_stage::StageContract;
pub use inference_stage::StageInput;
pub use inference_stage::StageOutput;
pub use inference_stage::StageSequence;
pub use inference_stage::StageTensor;
pub use torch_engine::GenerationStats;
pub use torch_engine::TextStream;
pub use torch_engine::TorchEngine;
pub use device_pool::DevicePool;
pub use kv_cache::CacheConfig;
pub use kv_cache::CacheOwner;
pub use kv_cache::KVCacheManager;
pub use kv_cache::KVCacheRegistry;
pub use kv_compat::check_compatibility;
pub use kv_compat::KvCompatDescriptor;
pub use kv_compat::KvCompatFingerprint;
pub use kv_compat::KvCompatMismatch;
pub use kv_compat::KvDtype;
pub use kv_compat::KvQuantMode;
pub use kv_compat::WeightIdentity;
pub use kv_compat::KV_COMPAT_FORMAT_VERSION;
pub use generation_metrics::GenerationMetricsAccumulator;
pub use generation_metrics::GenerationQualityMetrics;
pub use generation_metrics::SessionMetrics;

Modules§

architectures
Architecture-specific model implementations
batched_lora
Batched multi-tenant LoRA forward pass
device_pool
Multi-GPU device abstraction (DevicePool).
generation_metrics
Generation quality metrics for self-supervised training
image_utils
Image loading and preprocessing utilities for multimodal models
inference_profile
Deployment and scheduling contract for inference-engine instances.
inference_stage
Thread-local execution boundary for one contiguous decoder stage.
kv_cache
Key-Value cache implementation for efficient autoregressive generation
kv_compat
KV-cache compatibility fingerprint and rejection policy.
model_config
Unified model configuration management
model_factory
Model factory for creating models with unified configuration management
rope
Rotary Position Embedding (RoPE) implementation for tch-rs
template_engine
Jinja2 template engine for chat templates using minijinja
tensor_helpers
tensor_sampling
Device-agnostic tensor sampling for token generation
token_metrics
OpenTelemetry token-burn metering for inference.
tokenizer_config
Trait-based tokenizer configuration for different model architectures
torch_engine
PyTorch-based inference engine using tch-rs
torch_utils
Utilities for safe PyTorch operations with proper error handling
ttn_profile
TTN (Test-Time Naturalization) analysis pipeline for model layer profiling.
weight_provider
Weight provider for streaming large models

Structs§

Fp8ExecutionError
Recoverable FP8 execution failure exposed by the native model forward path.
MistralEngine

Enums§

KVQuantType
Generated from Cap’n Proto enum KVQuantType
ModelBuilderConfig

Traits§

RuntimeEngine
Core runtime engine trait - all engines implement this

Functions§

create_engine
Create the default runtime engine (now uses PyTorch)