Skip to main content

Module qwen

Module qwen 

Source
Expand description

Qwen model adapter that reuses Llama implementation with Qwen-specific configurations

Qwen models are architecturally similar to Llama but with key differences:

  • RoPE base (rope_theta) varies by model and version - always read from config.json
    • Qwen2.5 models typically use 1,000,000
    • Qwen3 models typically use 10,000 (standard)
  • Context window varies by model (read from max_position_embeddings in config.json)
    • Qwen3-4B: 40,960 tokens
    • Can extend via YARN scaling for longer context in some variants
  • Vocabulary size of 151,936 tokens (Qwen2/Qwen3)
  • SiLU activation function (same as Llama)
  • GQA (Grouped-Query Attention) used in Qwen3 models (num_key_value_heads < num_attention_heads)

Structsยง

QwenAdapter
Create a Qwen model by adapting Llama implementation