Expand description
PyTorch-based inference engine using tch-rs
Structs§
- Context
State - Basic context state for tracking generation state
- Generation
Stats - Statistics about text generation with separate prefill/inference metrics
- Text
Stream - Stream that yields decoded UTF-8 text chunks during generation.
- Torch
Engine - PyTorch inference engine using tch-rs
Functions§
- resolve_
inference_ compute_ kind - Resolve the compute dtype independently from checkpoint weight storage.
This pure policy accepts the documented case-insensitive precision aliases;
it performs no device allocation.
fp8retains legacy BF16 compute except for qualified channel-FP8 CPU checkpoints, whose activation QDQ/matmuls run in Float32. It never requests hardware Float8 activations on CPU. Serving adapters must still enforce their own qualified-device/dtype gate.