Skip to main content

assign_anchor_token

Function assign_anchor_token 

Source
pub fn assign_anchor_token(tokenizer: &mut Tokenizer) -> Result<u32, Error>
Expand description

Assign the learned anchor token id in tokenizer, adding it as a special token when absent. Idempotent: an existing entry (e.g. loaded from a converted bundle’s tokenizer) is reused, so the id is stable across encode sites that share the bundle.

The assigned id is part of the model bundle recorded at conversion time (P1.2); the freshly extended backbone embedding row for this id is what “learned anchor” refers to — the token starts untrained and is learned during distillation (P1.4).