Expand description
Tokenizer-bound linearization: the canonical jev-1 serialization (P0.1a) turned into token ids plus a per-question anchor map, with the 2048-token spec budget enforced against the real tokenizer and the learned anchor token id assigned.
Structs§
- Decision
Encoder - The question/criteria encoder. Wraps the bundle tokenizer with the assigned anchor id and enforces the tokenizer-bound spec budget.
- Encoded
Request - One tokenized decision request: the ids fed to the backbone and the anchor map the shared scorer reads at.
- Question
Anchors - Per-question anchor readout positions in one encoded request.
Constants§
- ANCHOR_
TOKEN - The learned special anchor token string. Byte-identical to
[
serialize::ANCHOR_PLACEHOLDER]: the placeholder in the frozen serialization is the special token’s text, so the tokenizer’s added-token matcher maps every placeholder occurrence to the anchor id.
Functions§
- assign_
anchor_ token - Assign the learned anchor token id in
tokenizer, adding it as a special token when absent. Idempotent: an existing entry (e.g. loaded from a converted bundle’s tokenizer) is reused, so the id is stable across encode sites that share the bundle.