Skip to main content

Module kv_compat

Module kv_compat 

Source
Expand description

KV-cache compatibility fingerprint and rejection policy.

Epic #1276, issue #1277.

KV states are a pure function of (a) the weights that produced them and (b) every model/runtime parameter that shapes the attention computation. Reusing KV produced under one configuration inside a request running under another is a silent correctness bug — the cached keys/values belong to a different function. This module defines the authoritative identity of that function as a versioned KvCompatDescriptor, reduces it to a stable content-addressed KvCompatFingerprint, and exposes a check_compatibility rejection policy: a cache whose descriptor differs in any authoritative field is rejected (its KV is discarded and recomputed).

This is the foundational lane item — every later reusable/durable lookup (cross-session prefix index, snapshot codec, durable store) consumes the fingerprint defined here. The module deliberately owns no tch/Cap’n Proto dependency: it is pure data + hashing, so it is fully unit-testable without a loaded model. Boundary types (KVQuantType, ModelConfig) are converted into the self-contained enums below at the engine seam.

§What participates (authoritative inputs)

Every field of KvCompatDescriptor is derived from the model/runtime state that actually produced the KV tensors — never from a human label:

  • Effective weights — base-weight content digest + adapter/delta generation (the live lora_generation actually applied to the request).
  • Model identity — name, architecture (model_type), config version.
  • Attention geometry — layers, KV heads, attention heads, head dim, hidden size (determines K/V tensor shape).
  • Effective attention behavior — QK-norm, partial-rotary dimension, per-layer local/global layout (layer_types), and hybrid/linear-attention dimensions (change K/V values, not just shape).
  • Position encoding — RoPE theta, RoPE scaling, max position embeddings (determines K values post-RoPE).
  • Compute dtype — the dtype the model computes/stores K/V in.
  • KV storage encoding — quantization mode in effect.
  • Block geometry — BLOCK_SIZE for paged caches.
  • Context cap — the effective max context actually bounding the cache.
  • Tokenizer identity — vocab size + a digest of the tokenizer bytes (cached token ids are only meaningful relative to one tokenizer).
  • Format version — the wire/layout version of the fingerprint itself.

§Authority and fail-closed reuse

A descriptor is only usable to bless a cache when every authoritative axis is populated (KvCompatDescriptor::is_authoritatively_complete): the base-weight content digest and the tokenizer hash. If the loader cannot obtain the base digest, or tokenizer hashing fails, the engine declines to mint an expected identity and the cache boundary fails closed — KV is recomputed rather than reused under a guess (decide_cache_reuse). An unstamped but populated cache is likewise discarded, never silently adopted.

§Relation to TTT delta invalidation

The registry’s existing push-based invalidate_for_tenant clears a tenant’s caches when that tenant’s LoRA delta changes. The fingerprint is the complementary pull-based guard checked at reuse time; it does not replace the push path. The descriptor carries an adapter_generation slot that a live counter (issue #1260) will feed so the pull path also covers per-request delta changes; until then, delta staleness is handled by the push path, and the fingerprint covers every other axis.

Structs§

KvCompatDescriptor
The complete set of authoritative inputs that determine whether KV states computed under one configuration can be reused under another.
KvCompatFingerprint
Content-addressed digest (SHA-256) of a KvCompatDescriptor.
MoeRouting
Mixture-of-experts routing configuration. Different routing changes which expert FFN each token traverses, and therefore the residual — and thus later K/V — feeding downstream attention. A dense model (the default) is all-zero.
RopeScalingFp
RoPE scaling config, with the factor canonicalized to its IEEE-754 bit pattern so the descriptor is byte-stable (no f32 equality pitfalls).
WeightIdentity
Authoritative identity of the effective weights (base + adapter/delta) that produced a cache’s KV.

Enums§

CacheReuseDecision
What to do with a cache encountered at the reuse boundary, produced by decide_cache_reuse. The policy is fail-closed: it never blesses a cache whose KV cannot be proven compatible with an authoritative identity.
KvCompatMismatch
Why a cache was rejected for reuse, naming the first differing authoritative field in a fixed comparison order. Produced by check_compatibility.
KvDtype
Compute dtype the model stores K/V in.
KvQuantMode
KV-cache storage encoding / quantization mode.

Constants§

KV_BLOCK_SIZE_DEFAULT
Default tokens-per-block for paged KV storage.
KV_COMPAT_FORMAT_VERSION
Wire/layout format version of the compatibility fingerprint.

Functions§

check_compatibility
Rejection policy: a cache whose descriptor differs in any authoritative field is rejected.
decide_cache_reuse
Decide whether a cache may be reused, given the authoritative expected identity (if any), the stamp currently on the cache (if any), and whether the cache provably carries no KV.
shard_content_digest
SHA-256 of a single shard’s resolved content bytes.