Skip to main content

LlamaModel

Struct LlamaModel 

Source
pub struct LlamaModel { /* private fields */ }
Expand description

Llama model implementation

Implementations§

Source§

impl LlamaModel

Source

pub fn from_safetensors( path: &Path, device: &Device, dtype: DType, ) -> Result<Self>

Create Llama model from SafeTensors

Source

pub fn from_weights( weights: &mut HashMap<String, Tensor>, device: &Device, dtype: DType, ) -> Result<Self>

Create Llama model from pre-loaded weight tensors

Source

pub fn from_weights_with_config( weights: &mut HashMap<String, Tensor>, config: LlamaConfig, device: &Device, dtype: DType, kv_quant_type: KVQuantType, ) -> Result<Self>

Create Llama model with explicit config (allows Qwen models to override) Build model from weights, taking mutable reference to free tensors as they’re processed. This reduces peak memory by ~50% by removing original weight tensors after transposing.

Source

pub fn stage_from_weights_with_config( weights: &mut HashMap<String, Tensor>, config: LlamaConfig, devices: &LayerDeviceMap, layer_range: Range<usize>, dtype: DType, kv_quant_type: KVQuantType, ) -> Result<Self>

Build a single pipeline stage of a Llama model (#314, 2b layer-split).

Loads only global decoder layers [layer_range.start..layer_range.end) onto their mapped devices (devices.device_for(g)), and gates the non-layer weights by stage position so middle stages carry none (M-LOAD seam #1):

  • is_first (layer_range.start == 0) → keep embed_tokens.
  • is_last (layer_range.end == devices.len()) → keep norm + lm_head (or the tied transpose).

weights must already contain (at least) this stage’s tensors. Production stage requests use ModelFactory::create_stage, whose authoritative HF-index loader supplies only the requested range and role tensors. Tensors not on the target device are moved with a single .to() at construction — the only placement cost; the forward path then does zero intra-stage copies.

State sizing follows the owned layer count, not the global count (M-LOAD seam #2): the KV-cache manager and self.layers are both sized to layer_range.len(), and the forward loop indexes them locally.

Source

pub fn detect_config_from_weights( weights: &HashMap<String, Tensor>, ) -> Result<LlamaConfig>

Detect configuration from weight tensor shapes

Source

pub fn parse_config(json_str: &str) -> Result<LlamaConfig>

Parse configuration from JSON

Trait Implementations§

Source§

impl ModelOperations for LlamaModel

Source§

fn architecture(&self) -> ModelArchitecture

Get the architecture type
Source§

fn config(&self) -> &dyn ArchitectureConfig

Get architecture configuration
Source§

fn forward(&self, input: &Tensor, _past_kv: Option<&Tensor>) -> Result<Tensor>

Forward pass through the model
Source§

fn var_store(&self) -> Option<&VarStore>

Get VarStore for training (if available)
Source§

fn var_store_mut(&mut self) -> Option<&mut VarStore>

Get mutable VarStore for training (if available)
Source§

fn forward_with_cache(&self, input: &Tensor, start_pos: usize) -> Result<Tensor>

Forward pass with position information for KV caching
Source§

fn forward_with_cache_and_delta( &self, input: &Tensor, start_pos: usize, delta: Option<&TenantDelta>, ) -> Result<Tensor>

Forward pass with KV caching and optional per-tenant delta injection Read more
Source§

fn forward_batched( &self, sequences: &mut [(Vec<i64>, usize, Arc<Mutex<KVCacheManager>>)], delta: Option<&TenantDelta>, ) -> Result<Tensor>

Batched ragged forward for continuous decode (#329, epic #310). Read more
Source§

fn embed_tokens(&self, input_ids: &Tensor) -> Result<Tensor>

Get token embeddings for input IDs
Source§

fn forward_from_embeddings( &self, embeddings: &Tensor, start_pos: usize, ) -> Result<Tensor>

Forward pass from pre-computed embeddings (for multimodal models) Read more
Source§

fn reshape_for_attention( &self, tensor: &Tensor, is_key_value: bool, ) -> Result<Tensor>

Reshape tensor for attention computation
Source§

fn decode_layer( &self, layer_idx: usize, hidden_states: &Tensor, _attention_mask: Option<&Tensor>, position_ids: Option<&Tensor>, _past_kv: Option<&LayerKVCache>, ) -> Result<(Tensor, Option<(Tensor, Tensor)>)>

Decode a single layer (for layer-wise processing)
Source§

fn decode_layer_with_delta( &self, layer_idx: usize, hidden_states: &Tensor, _attention_mask: Option<&Tensor>, position_ids: Option<&Tensor>, _past_kv: Option<&LayerKVCache>, delta: &TenantDelta, ) -> Result<(Tensor, Option<(Tensor, Tensor)>)>

Decode a single layer with per-tenant delta injection (for TTT training path) Read more
Source§

fn forward_layers( &self, hidden: &Tensor, range: Range<usize>, start_pos: usize, delta: Option<&TenantDelta>, ) -> Result<Tensor>

Run a contiguous range of decoder layers — the 2b intra-host pipeline (layer-split) primitive (#314). Read more
Source§

fn forward_layers_train( &self, hidden: &Tensor, range: Range<usize>, delta: Option<&TenantDelta>, ) -> Result<Tensor>

Training-path sibling of Self::forward_layers — the cross-device autograd primitive for TTT-on-split (#316, M-TRAIN-COUPLING). Read more
Source§

fn apply_final_norm(&self, hidden_states: &Tensor) -> Result<Tensor>

Apply final layer normalization
Source§

fn lm_head(&self, hidden_states: &Tensor) -> Result<Tensor>

Apply language model head to get logits
Source§

fn num_layers(&self) -> usize

Get the number of transformer layers
Source§

fn apply_rope(&self, _tensor: &Tensor, _position_ids: &Tensor) -> Result<Tensor>

Apply RoPE (Rotary Position Embeddings) if supported
Source§

fn normalize(&self, tensor: &Tensor) -> Result<Tensor>

Apply architecture-specific normalization (RMSNorm, LayerNorm, etc.)
Source§

fn get_attention_mask( &self, seq_len: usize, past_kv_len: usize, ) -> Result<Tensor>

Get attention mask for the architecture
Source§

fn as_any(&self) -> &dyn Any

Downcast to Any for accessing architecture-specific methods
Source§

fn clear_kv_cache(&self)

Clear the KV cache (for new generation)
Source§

fn kv_cache_memory_usage(&self) -> usize

Get KV cache memory usage in bytes
Source§

fn set_kv_cache(&mut self, cache: Arc<Mutex<KVCacheManager>>)

Set an external KV cache (for session-based cache management) Read more
Source§

fn get_kv_cache(&self) -> Option<Arc<Mutex<KVCacheManager>>>

Get the current KV cache (for external management)
Source§

fn take_kv_cache(&mut self) -> Option<Arc<Mutex<KVCacheManager>>>

Take ownership of the KV cache (removes it from the model)
Source§

fn get_tokenizer_config(&self) -> Box<dyn TokenizerConfig>

Get the tokenizer configuration for this model Read more
Source§

fn decision_hidden_states(&self, _input: &Tensor) -> Result<Tensor>

Stateless full-sequence final-normalized hidden states; no vocabulary logits. Implementations must use fresh call-local recurrent/KV state.
Source§

fn decision_hidden_states_checked( &self, input: &Tensor, check: &dyn Fn() -> Result<()>, ) -> Result<Tensor>

Stateless decision forward with a cooperative deadline/cancellation check. Native implementations call the checker between decoder layers.
Source§

fn decision_output_embedding_rows(&self, _ids: &[i64]) -> Result<Tensor>

Gather original output embedding rows, preserving the LM-head vocabulary axis.
Source§

fn is_multimodal(&self) -> bool

Check if this model is multimodal
Source§

fn prepare_multimodal_inputs( &self, _input_ids: &Tensor, _pixel_values: Option<&Tensor>, _images_seq_mask: Option<&Tensor>, _images_emb_mask: Option<&Tensor>, ) -> Result<Tensor>

Prepare inputs with vision embeddings (for multimodal models) Read more
Source§

fn encode_vision(&self, _pixel_values: &Tensor) -> Result<Tensor>

Encode images through the vision encoder and return embeddings. Read more
Source§

impl Send for LlamaModel

Source§

impl Sync for LlamaModel

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<T> AnySync for T
where T: Any + Send + Sync,

§

fn into_any_arc(self: Arc<T>) -> Arc<dyn Any + Sync + Send> ⓘ

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSync for T
where T: Sync,

§

impl<T> MaybeSync for T
where T: Sync,