Skip to main content

Module torch_engine

Module torch_engine 

Source
Expand description

PyTorch-based inference engine using tch-rs

Structs§

ContextState
Basic context state for tracking generation state
GenerationStats
Statistics about text generation with separate prefill/inference metrics
TextStream
Stream that yields decoded UTF-8 text chunks during generation.
TorchEngine
PyTorch inference engine using tch-rs

Functions§

resolve_inference_compute_kind
Resolve the compute dtype independently from checkpoint weight storage. This pure policy accepts the documented case-insensitive precision aliases; it performs no device allocation. fp8 retains legacy BF16 compute except for qualified channel-FP8 CPU checkpoints, whose activation QDQ/matmuls run in Float32. It never requests hardware Float8 activations on CPU. Serving adapters must still enforce their own qualified-device/dtype gate.