Expand description
Key-Value cache implementation for efficient autoregressive generation
This module implements KV caching to avoid recomputing past key and value states during inference, providing 10-50x speedup for long sequences.
§Memory Optimization
The cache uses a chunked growth strategy instead of pre-allocating the full max_seq_len. This saves ~4GB+ VRAM for models with large context windows.
§Quantization Support
Supports optional quantization via bitsandbytes:
None: Full precision (FP16/BF16) - defaultInt8: 8-bit quantizationNf4: 4-bit NormalFloat (best quality for 4-bit)Fp4: 4-bit FloatingPoint
Quantization uses a hybrid approach:
- Historical tokens are stored quantized (memory savings)
- Recent tokens stay in a full-precision buffer (fast updates)
- Dequantized view is cached across layers (avoids repeated dequantization)
Structs§
- Block
Pool - Shared pool of fixed-size GPU tensor blocks for KV cache storage.
- Block
Pool Exhausted - Failure to allocate a block from the shared pool because no free blocks remain.
- Cache
Config - Configuration for KV cache instances
- KVCache
Manager - KV cache manager for all layers in a model
- KVCache
Registry - Registry managing multiple KV cache instances for concurrent sessions.
- LayerKV
Cache - KV cache for a single attention layer
Enums§
- Cache
Location - Where a KV cache’s tensors are currently stored.
- Cache
Owner - Identifies the owner of a KV cache instance for session isolation.
Constants§
- BLOCK_
SIZE - Number of tokens per block in the paged KV cache. 256 balances fragmentation overhead vs block reuse granularity.
Type Aliases§
- BlockId
- Opaque handle to a block in the pool.