Skip to main content

Module architectures

Module architectures 

Source
Expand description

Architecture-specific model implementations

This module provides abstractions and implementations for different transformer architectures (Llama, Gemma, Qwen, etc.) with proper tensor shape handling and LoRA compatibility.

Modules§

config
Architecture configuration traits and implementations
gemma
Gemma model implementation with Multi-Query Attention support
janus
Janus multimodal vision-language model implementation
llama
Llama model implementation with support for Llama 1/2/3 and GQA
qwen
Qwen model adapter that reuses Llama implementation with Qwen-specific configurations
qwen3_5
Qwen3.5 hybrid SSM/full-attention model implementation
qwen3_5_vision
Qwen3.5 vision encoder
siglip
SigLIP (Sigmoid Loss for Language-Image Pre-training) Vision Transformer

Structs§

AttentionConfig
Attention configuration

Enums§

AttentionType
Attention mechanism types
ModelArchitecture
Supported model architectures
VisionEncoderType
Types of vision encoders for multimodal models

Traits§

ArchitectureConfig
Core configuration for model architectures
ModelOperations
Core trait for architecture-specific model operations

Functions§

add_delta
Add a TTT/LoRA delta correction to a base activation, coercing the correction to the base tensor’s dtype first.