Expand description
Qwen model adapter that reuses Llama implementation with Qwen-specific configurations
Qwen models are architecturally similar to Llama but with key differences:
- RoPE base (rope_theta) varies by model and version - always read from config.json
- Qwen2.5 models typically use 1,000,000
- Qwen3 models typically use 10,000 (standard)
- Context window varies by model (read from max_position_embeddings in config.json)
- Qwen3-4B: 40,960 tokens
- Can extend via YARN scaling for longer context in some variants
- Vocabulary size of 151,936 tokens (Qwen2/Qwen3)
- SiLU activation function (same as Llama)
- GQA (Grouped-Query Attention) used in Qwen3 models (num_key_value_heads < num_attention_heads)
Structsยง
- Qwen
Adapter - Create a Qwen model by adapting Llama implementation