Skip to main content

Module architectures

Module architectures 

Source
Expand description

Architecture-specific model implementations

This module provides abstractions and implementations for different transformer architectures (Llama, Gemma, Qwen, etc.) with proper tensor shape handling and LoRA compatibility.

Re-exports§

pub use config::ArchitectureConfig;
pub use config::AttentionConfig;

Modules§

config
Architecture configuration traits and implementations
gemma
Gemma model implementation with Multi-Query Attention support
janus
Janus multimodal vision-language model implementation
llama
Llama model implementation with support for Llama 1/2/3 and GQA
qwen
Qwen model adapter that reuses Llama implementation with Qwen-specific configurations
qwen3_5
Qwen3.5 hybrid SSM/full-attention model implementation
qwen3_5_vision
Qwen3.5 vision encoder
siglip
SigLIP (Sigmoid Loss for Language-Image Pre-training) Vision Transformer

Enums§

AttentionType
Attention mechanism types
ModelArchitecture
Supported model architectures
VisionEncoderType
Types of vision encoders for multimodal models

Traits§

ModelOperations
Core trait for architecture-specific model operations

Functions§

add_delta
Add a TTT/LoRA delta correction to a base activation, coercing the correction to the base tensor’s dtype first.