Expand description
KV-cache compatibility fingerprint and rejection policy.
KV states are a pure function of (a) the weights that produced them and
(b) every model/runtime parameter that shapes the attention computation.
Reusing KV produced under one configuration inside a request running under
another is a silent correctness bug — the cached keys/values belong to a
different function. This module defines the authoritative identity of
that function as a versioned KvCompatDescriptor, reduces it to a stable
content-addressed KvCompatFingerprint, and exposes a check_compatibility
rejection policy: a cache whose descriptor differs in any authoritative
field is rejected (its KV is discarded and recomputed).
This is the foundational lane item — every later reusable/durable lookup
(cross-session prefix index, snapshot codec, durable store) consumes the
fingerprint defined here. The module deliberately owns no tch/Cap’n
Proto dependency: it is pure data + hashing, so it is fully unit-testable
without a loaded model. Boundary types (KVQuantType, ModelConfig) are
converted into the self-contained enums below at the engine seam.
§What participates (authoritative inputs)
Every field of KvCompatDescriptor is derived from the model/runtime state
that actually produced the KV tensors — never from a human label:
- Effective weights — base-weight content digest + adapter/delta
generation (the live
lora_generationactually applied to the request). - Model identity — name, architecture (
model_type), config version. - Attention geometry — layers, KV heads, attention heads, head dim, hidden size (determines K/V tensor shape).
- Effective attention behavior — QK-norm, partial-rotary dimension,
per-layer local/global layout (
layer_types), and hybrid/linear-attention dimensions (change K/V values, not just shape). - Position encoding — RoPE theta, RoPE scaling, max position embeddings (determines K values post-RoPE).
- Compute dtype — the dtype the model computes/stores K/V in.
- KV storage encoding — quantization mode in effect.
- Block geometry —
BLOCK_SIZEfor paged caches. - Context cap — the effective max context actually bounding the cache.
- Tokenizer identity — vocab size + a digest of the tokenizer bytes (cached token ids are only meaningful relative to one tokenizer).
- Format version — the wire/layout version of the fingerprint itself.
§Authority and fail-closed reuse
A descriptor is only usable to bless a cache when every authoritative axis is
populated (KvCompatDescriptor::is_authoritatively_complete): the
base-weight content digest and the tokenizer hash. If the loader cannot
obtain the base digest, or tokenizer hashing fails, the engine declines to
mint an expected identity and the cache boundary fails closed — KV is
recomputed rather than reused under a guess (decide_cache_reuse). An
unstamped but populated cache is likewise discarded, never silently adopted.
§Relation to TTT delta invalidation
The registry’s existing push-based invalidate_for_tenant
clears a tenant’s caches when that tenant’s LoRA delta changes. The
fingerprint is the complementary pull-based guard checked at reuse time;
it does not replace the push path. The descriptor carries an
adapter_generation slot that a live counter (issue #1260) will feed so the
pull path also covers per-request delta changes; until then, delta staleness
is handled by the push path, and the fingerprint covers every other axis.
Structs§
- KvCompat
Descriptor - The complete set of authoritative inputs that determine whether KV states computed under one configuration can be reused under another.
- KvCompat
Fingerprint - Content-addressed digest (SHA-256) of a
KvCompatDescriptor. - MoeRouting
- Mixture-of-experts routing configuration. Different routing changes which expert FFN each token traverses, and therefore the residual — and thus later K/V — feeding downstream attention. A dense model (the default) is all-zero.
- Rope
Scaling Fp - RoPE scaling config, with the factor canonicalized to its IEEE-754 bit
pattern so the descriptor is byte-stable (no
f32equality pitfalls). - Weight
Identity - Authoritative identity of the effective weights (base + adapter/delta) that produced a cache’s KV.
Enums§
- Cache
Reuse Decision - What to do with a cache encountered at the reuse boundary, produced by
decide_cache_reuse. The policy is fail-closed: it never blesses a cache whose KV cannot be proven compatible with an authoritative identity. - KvCompat
Mismatch - Why a cache was rejected for reuse, naming the first differing authoritative
field in a fixed comparison order. Produced by
check_compatibility. - KvDtype
- Compute dtype the model stores K/V in.
- KvQuant
Mode - KV-cache storage encoding / quantization mode.
Constants§
- KV_
BLOCK_ SIZE_ DEFAULT - Default tokens-per-block for paged KV storage.
- KV_
COMPAT_ FORMAT_ VERSION - Wire/layout format version of the compatibility fingerprint.
Functions§
- check_
compatibility - Rejection policy: a cache whose descriptor differs in any authoritative field is rejected.
- decide_
cache_ reuse - Decide whether a cache may be reused, given the authoritative expected identity (if any), the stamp currently on the cache (if any), and whether the cache provably carries no KV.
- shard_
content_ digest - SHA-256 of a single shard’s resolved content bytes.