pub struct LlamaConfig {Show 20 fields
pub version: u8,
pub num_attention_heads: u32,
pub num_key_value_heads: u32,
pub hidden_size: u32,
pub head_dim: u32,
pub intermediate_size: u32,
pub max_position_embeddings: u32,
pub rms_norm_eps: f32,
pub vocab_size: u32,
pub original_vocab_size: u32,
pub num_hidden_layers: u32,
pub rope_theta: f32,
pub rope_scaling: Option<RopeScaling>,
pub hidden_activation: String,
pub query_pre_attn_scalar: Option<f32>,
pub use_qk_norm: bool,
pub scale_embeddings: bool,
pub layer_types: Vec<String>,
pub rope_local_base_freq: Option<f32>,
pub model_architecture: ModelArchitecture,
}Expand description
Llama model configuration
Fields§
§version: u8Llama version (1, 2, or 3)
num_attention_heads: u32Number of attention heads for queries
num_key_value_heads: u32Number of key-value heads (for GQA in Llama 2/3)
Hidden dimension size
head_dim: u32Head dimension
intermediate_size: u32Intermediate size for FFN
max_position_embeddings: u32Maximum position embeddings
rms_norm_eps: f32RMSNorm epsilon
vocab_size: u32Vocabulary size (may be padded)
original_vocab_size: u32Original vocabulary size (before padding)
Number of hidden layers
rope_theta: f32RoPE theta
rope_scaling: Option<RopeScaling>RoPE scaling (for Llama 3)
Hidden activation function (silu for Llama, gelu_pytorch_tanh for Gemma3)
query_pre_attn_scalar: Option<f32>Query pre-attention scalar for QK-norm (Gemma3)
use_qk_norm: boolUse QK-norm (Gemma3)
scale_embeddings: boolScale embeddings by sqrt(hidden_size) (Gemma3)
layer_types: Vec<String>Layer types for sliding window attention (Gemma3)
rope_local_base_freq: Option<f32>RoPE theta for local attention layers (Gemma3)
model_architecture: ModelArchitectureThe real model architecture this LlamaModel stands in for.
LlamaModel is reused as the runtime for several architecturally-llama
families (Llama, Qwen, Mistral). The tokenizer-config dispatch keys on
ModelOperations::architecture(), so a Qwen3 checkpoint loaded via
create_qwen_model must report Qwen here — otherwise the
QwenTokenizerConfig (which pads the tokenizer vocab up to the model’s
embedding size) is never applied and the tokenizer/embedding vocab
mismatch silently causes a CUDA index-out-of-bounds on the embedding
lookup (#143). Defaults to Llama; create_qwen_model overrides it.
Implementations§
Source§impl LlamaConfig
impl LlamaConfig
Sourcepub fn llama3_70b() -> Self
pub fn llama3_70b() -> Self
Create config for Llama 3 70B
Trait Implementations§
Source§impl ArchitectureConfig for LlamaConfig
impl ArchitectureConfig for LlamaConfig
Source§fn num_attention_heads(&self) -> usize
fn num_attention_heads(&self) -> usize
Source§fn num_key_value_heads(&self) -> usize
fn num_key_value_heads(&self) -> usize
Source§fn intermediate_size(&self) -> usize
fn intermediate_size(&self) -> usize
Source§fn vocab_size(&self) -> usize
fn vocab_size(&self) -> usize
Source§fn max_position_embeddings(&self) -> usize
fn max_position_embeddings(&self) -> usize
Source§fn rope_theta(&self) -> Option<f32>
fn rope_theta(&self) -> Option<f32>
Source§fn layer_norm_eps(&self) -> f32
fn layer_norm_eps(&self) -> f32
Source§fn use_rms_norm(&self) -> bool
fn use_rms_norm(&self) -> bool
Source§fn use_parallel_residual(&self) -> bool
fn use_parallel_residual(&self) -> bool
Source§fn attention_type(&self) -> AttentionType
fn attention_type(&self) -> AttentionType
Source§fn default_dtype(&self) -> DType
fn default_dtype(&self) -> DType
Source§impl Clone for LlamaConfig
impl Clone for LlamaConfig
Source§fn clone(&self) -> LlamaConfig
fn clone(&self) -> LlamaConfig
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreSource§impl Debug for LlamaConfig
impl Debug for LlamaConfig
Auto Trait Implementations§
impl Freeze for LlamaConfig
impl RefUnwindSafe for LlamaConfig
impl Send for LlamaConfig
impl Sync for LlamaConfig
impl Unpin for LlamaConfig
impl UnsafeUnpin for LlamaConfig
impl UnwindSafe for LlamaConfig
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more