Skip to main content

Module kv_cache

Module kv_cache 

Source
Expand description

Key-Value cache implementation for efficient autoregressive generation

This module implements KV caching to avoid recomputing past key and value states during inference, providing 10-50x speedup for long sequences.

§Memory Optimization

The cache uses a chunked growth strategy instead of pre-allocating the full max_seq_len. This saves ~4GB+ VRAM for models with large context windows.

§Quantization Support

Supports optional quantization via bitsandbytes:

  • None: Full precision (FP16/BF16) - default
  • Int8: 8-bit quantization
  • Nf4: 4-bit NormalFloat (best quality for 4-bit)
  • Fp4: 4-bit FloatingPoint

Quantization uses a hybrid approach:

  • Historical tokens are stored quantized (memory savings)
  • Recent tokens stay in a full-precision buffer (fast updates)
  • Dequantized view is cached across layers (avoids repeated dequantization)

Structs§

BlockPool
Shared pool of fixed-size GPU tensor blocks for KV cache storage.
BlockPoolExhausted
Failure to allocate a block from the shared pool because no free blocks remain.
CacheConfig
Configuration for KV cache instances
KVCacheManager
KV cache manager for all layers in a model
KVCacheRegistry
Registry managing multiple KV cache instances for concurrent sessions.
LayerKVCache
KV cache for a single attention layer

Enums§

CacheLocation
Where a KV cache’s tensors are currently stored.
CacheOwner
Identifies the owner of a KV cache instance for session isolation.

Constants§

BLOCK_SIZE
Number of tokens per block in the paged KV cache. 256 balances fragmentation overhead vs block reuse granularity.

Type Aliases§

BlockId
Opaque handle to a block in the pool.