Expand description
Architecture-specific model implementations
This module provides abstractions and implementations for different transformer architectures (Llama, Gemma, Qwen, etc.) with proper tensor shape handling and LoRA compatibility.
Modules§
- config
- Architecture configuration traits and implementations
- gemma
- Gemma model implementation with Multi-Query Attention support
- janus
- Janus multimodal vision-language model implementation
- llama
- Llama model implementation with support for Llama 1/2/3 and GQA
- qwen
- Qwen model adapter that reuses Llama implementation with Qwen-specific configurations
- qwen3_5
- Qwen3.5 hybrid SSM/full-attention model implementation
- qwen3_
5_ vision - Qwen3.5 vision encoder
- siglip
- SigLIP (Sigmoid Loss for Language-Image Pre-training) Vision Transformer
Structs§
- Attention
Config - Attention configuration
Enums§
- Attention
Type - Attention mechanism types
- Model
Architecture - Supported model architectures
- Vision
Encoder Type - Types of vision encoders for multimodal models
Traits§
- Architecture
Config - Core configuration for model architectures
- Model
Operations - Core trait for architecture-specific model operations
Functions§
- add_
delta - Add a TTT/LoRA delta correction to a base activation, coercing the correction to the base tensor’s dtype first.