pub trait TokenizerConfig: Send + Sync {
// Required method
fn configure_tokenizer(
&self,
tokenizer: &mut Tokenizer,
model_vocab_size: usize,
) -> Result<(), Error>;
// Provided methods
fn get_special_tokens(
&self,
_model_vocab_size: usize,
_tokenizer_vocab_size: usize,
) -> Vec<AddedToken> { ... }
fn is_special_token(&self, _token_id: usize, _vocab_size: usize) -> bool { ... }
fn format_special_token(
&self,
_token_id: usize,
_vocab_size: usize,
) -> Option<String> { ... }
fn decode_token(
&self,
tokenizer: &Tokenizer,
token_id: u32,
) -> Result<Option<String>, Error> { ... }
}Expand description
Trait for model-specific tokenizer configuration
Each model architecture can implement this trait to provide custom tokenizer configuration, including adding special tokens, configuring post-processors, and handling vocabulary mismatches.
Required Methods§
Sourcefn configure_tokenizer(
&self,
tokenizer: &mut Tokenizer,
model_vocab_size: usize,
) -> Result<(), Error>
fn configure_tokenizer( &self, tokenizer: &mut Tokenizer, model_vocab_size: usize, ) -> Result<(), Error>
Configure the tokenizer for this specific model architecture
This method is called after loading the base tokenizer from file, allowing the model to add special tokens, configure post-processing, or make other model-specific adjustments.
§Arguments
tokenizer- The tokenizer to configure (mutable reference)model_vocab_size- The vocabulary size from the model’s config
§Returns
Result<()>- Success or error during configuration
Provided Methods§
Sourcefn get_special_tokens(
&self,
_model_vocab_size: usize,
_tokenizer_vocab_size: usize,
) -> Vec<AddedToken>
fn get_special_tokens( &self, _model_vocab_size: usize, _tokenizer_vocab_size: usize, ) -> Vec<AddedToken>
Get special tokens that should be added for this model
This is called by configure_tokenizer but can also be used
independently if needed.
§Arguments
model_vocab_size- Total vocabulary size expected by the modeltokenizer_vocab_size- Current size of the tokenizer vocabulary
§Returns
- Vector of
AddedTokenobjects to add to the tokenizer
Sourcefn is_special_token(&self, _token_id: usize, _vocab_size: usize) -> bool
fn is_special_token(&self, _token_id: usize, _vocab_size: usize) -> bool
Sourcefn format_special_token(
&self,
_token_id: usize,
_vocab_size: usize,
) -> Option<String>
fn format_special_token( &self, _token_id: usize, _vocab_size: usize, ) -> Option<String>
Get display representation for out-of-vocabulary or special tokens
This is useful for debugging or when special tokens need custom display.
§Arguments
token_id- The token ID to formatvocab_size- The base vocabulary size
§Returns
Some(String)with the formatted token, orNoneif no special formatting
Sourcefn decode_token(
&self,
tokenizer: &Tokenizer,
token_id: u32,
) -> Result<Option<String>, Error>
fn decode_token( &self, tokenizer: &Tokenizer, token_id: u32, ) -> Result<Option<String>, Error>
Decode a token that might be out of vocabulary
This method is called when a token needs to be decoded. It can handle tokens that are beyond the tokenizer’s vocabulary by returning a special representation.
§Arguments
tokenizer- The tokenizer to use for decodingtoken_id- The token ID to decode
§Returns
Ok(Some(String))- Successfully decoded token or special representationOk(None)- Token should be skippedErr- Decoding error