Expand description
Trait-based tokenizer configuration for different model architectures
This module provides a clean abstraction for model-specific tokenizer configuration, allowing each model architecture to customize tokenizer behavior without polluting the global runtime code.
Structs§
- Default
Tokenizer Config - Default tokenizer configuration for unknown model types
- Gemma
Tokenizer Config - Gemma model tokenizer configuration
- Llama
Tokenizer Config - Llama model tokenizer configuration
- Qwen
Tokenizer Config - Qwen model tokenizer configuration
Traits§
- Tokenizer
Config - Trait for model-specific tokenizer configuration