Skip to main content

Module tokenizer_config

Module tokenizer_config 

Source
Expand description

Trait-based tokenizer configuration for different model architectures

This module provides a clean abstraction for model-specific tokenizer configuration, allowing each model architecture to customize tokenizer behavior without polluting the global runtime code.

Structs§

DefaultTokenizerConfig
Default tokenizer configuration for unknown model types
GemmaTokenizerConfig
Gemma model tokenizer configuration
LlamaTokenizerConfig
Llama model tokenizer configuration
QwenTokenizerConfig
Qwen model tokenizer configuration

Traits§

TokenizerConfig
Trait for model-specific tokenizer configuration