Skip to main content

Module encoder

Module encoder 

Source
Expand description

Tokenizer-bound linearization: the canonical jev-1 serialization (P0.1a) turned into token ids plus a per-question anchor map, with the 2048-token spec budget enforced against the real tokenizer and the learned anchor token id assigned.

Structs§

DecisionEncoder
The question/criteria encoder. Wraps the bundle tokenizer with the assigned anchor id and enforces the tokenizer-bound spec budget.
EncodedRequest
One tokenized decision request: the ids fed to the backbone and the anchor map the shared scorer reads at.
QuestionAnchors
Per-question anchor readout positions in one encoded request.

Constants§

ANCHOR_TOKEN
The learned special anchor token string. Byte-identical to [serialize::ANCHOR_PLACEHOLDER]: the placeholder in the frozen serialization is the special token’s text, so the tokenizer’s added-token matcher maps every placeholder occurrence to the anchor id.

Functions§

assign_anchor_token
Assign the learned anchor token id in tokenizer, adding it as a special token when absent. Idempotent: an existing entry (e.g. loaded from a converted bundle’s tokenizer) is reused, so the id is stable across encode sites that share the bundle.