Skip to main content

TokenizerConfig

Trait TokenizerConfig 

Source
pub trait TokenizerConfig: Send + Sync {
    // Required method
    fn configure_tokenizer(
        &self,
        tokenizer: &mut Tokenizer,
        model_vocab_size: usize,
    ) -> Result<()>;

    // Provided methods
    fn get_special_tokens(
        &self,
        _model_vocab_size: usize,
        _tokenizer_vocab_size: usize,
    ) -> Vec<AddedToken> { ... }
    fn is_special_token(&self, _token_id: usize, _vocab_size: usize) -> bool { ... }
    fn format_special_token(
        &self,
        _token_id: usize,
        _vocab_size: usize,
    ) -> Option<String> { ... }
    fn decode_token(
        &self,
        tokenizer: &Tokenizer,
        token_id: u32,
    ) -> Result<Option<String>> { ... }
}
Expand description

Trait for model-specific tokenizer configuration

Each model architecture can implement this trait to provide custom tokenizer configuration, including adding special tokens, configuring post-processors, and handling vocabulary mismatches.

Required Methods§

Source

fn configure_tokenizer( &self, tokenizer: &mut Tokenizer, model_vocab_size: usize, ) -> Result<()>

Configure the tokenizer for this specific model architecture

This method is called after loading the base tokenizer from file, allowing the model to add special tokens, configure post-processing, or make other model-specific adjustments.

§Arguments
  • tokenizer - The tokenizer to configure (mutable reference)
  • model_vocab_size - The vocabulary size from the model’s config
§Returns
  • Result<()> - Success or error during configuration

Provided Methods§

Source

fn get_special_tokens( &self, _model_vocab_size: usize, _tokenizer_vocab_size: usize, ) -> Vec<AddedToken>

Get special tokens that should be added for this model

This is called by configure_tokenizer but can also be used independently if needed.

§Arguments
  • model_vocab_size - Total vocabulary size expected by the model
  • tokenizer_vocab_size - Current size of the tokenizer vocabulary
§Returns
  • Vector of AddedToken objects to add to the tokenizer
Source

fn is_special_token(&self, _token_id: usize, _vocab_size: usize) -> bool

Check if a token ID represents a special token for this model

§Arguments
  • token_id - The token ID to check
  • vocab_size - The base vocabulary size (without special tokens)
§Returns
  • true if this is a special token, false otherwise
Source

fn format_special_token( &self, _token_id: usize, _vocab_size: usize, ) -> Option<String>

Get display representation for out-of-vocabulary or special tokens

This is useful for debugging or when special tokens need custom display.

§Arguments
  • token_id - The token ID to format
  • vocab_size - The base vocabulary size
§Returns
  • Some(String) with the formatted token, or None if no special formatting
Source

fn decode_token( &self, tokenizer: &Tokenizer, token_id: u32, ) -> Result<Option<String>>

Decode a token that might be out of vocabulary

This method is called when a token needs to be decoded. It can handle tokens that are beyond the tokenizer’s vocabulary by returning a special representation.

§Arguments
  • tokenizer - The tokenizer to use for decoding
  • token_id - The token ID to decode
§Returns
  • Ok(Some(String)) - Successfully decoded token or special representation
  • Ok(None) - Token should be skipped
  • Err - Decoding error

Implementors§