pub struct Qwen3_5Model { /* private fields */ }Implementations§
Source§impl Qwen3_5Model
impl Qwen3_5Model
Sourcepub fn from_weights(
weights: &mut HashMap<String, Tensor>,
cfg: Qwen3_5TextConfig,
vision_cfg: Option<Qwen3_5VisionConfig>,
device: &Device,
dtype: Kind,
kv_quant_type: KVQuantType,
speculative_decoding: bool,
) -> Result<Qwen3_5Model, Error>
pub fn from_weights( weights: &mut HashMap<String, Tensor>, cfg: Qwen3_5TextConfig, vision_cfg: Option<Qwen3_5VisionConfig>, device: &Device, dtype: Kind, kv_quant_type: KVQuantType, speculative_decoding: bool, ) -> Result<Qwen3_5Model, Error>
Construct from caller-owned tensors. Disabled speculation discards any MTP tensors already supplied; use ModelFactory to filter them before materialization and avoid their temporary device allocation as well.
Sourcepub fn stage_from_weights_with_config(
weights: &mut HashMap<String, Tensor>,
cfg: Qwen3_5TextConfig,
devices: &LayerDeviceMap,
layer_range: Range<usize>,
dtype: Kind,
kv_quant_type: KVQuantType,
) -> Result<Qwen3_5Model, Error>
pub fn stage_from_weights_with_config( weights: &mut HashMap<String, Tensor>, cfg: Qwen3_5TextConfig, devices: &LayerDeviceMap, layer_range: Range<usize>, dtype: Kind, kv_quant_type: KVQuantType, ) -> Result<Qwen3_5Model, Error>
Build a single pipeline stage of a Qwen3.5 model (#314, 2b layer-split).
Mirrors LlamaModel::stage_from_weights_with_config, plus Qwen3.5’s
specifics:
- Hybrid layer type is chosen by the global layer index inside
Self::build_layer(cfg.layer_types[g]), so a shard owning[a..b)loads the correct GDN/full-attn mixer per global layer. - SSM state (
conv_states/rec_states) is sized to the OWNED layer count (layer_range.len()), indexed by the LOCAL layer index, and is stage-local — never transferred across a pipeline boundary (M-LOAD seam #2). Onlyhidden+start_poscross a boundary.
Non-layer weights are gated by stage position (M-LOAD seam #1):
is_first(layer_range.start == 0) → keepembed_tokens.is_last(layer_range.end == num_hidden_layers) → keepnorm+lm_head(or the tied transpose).
Each owned layer is placed on its mapped device (devices.device_for(g))
via the into_device builders — the only placement cost; the forward path
then performs zero intra-stage copies. Vision weights are out of scope for
a pipeline stage (text backbone split only).
Sourcepub fn text_config(&self) -> &Qwen3_5TextConfig
pub fn text_config(&self) -> &Qwen3_5TextConfig
Expose the Qwen3.5 text config for external callers (e.g., get_per_layer_lora_dims).
Sourcepub fn snapshot_ssm_states(&self) -> (Vec<Option<Tensor>>, Vec<Option<Tensor>>)
pub fn snapshot_ssm_states(&self) -> (Vec<Option<Tensor>>, Vec<Option<Tensor>>)
Deep-copy current SSM states for TTT snapshot/restore (C4 fix: must use .copy(), not .shallow_clone()).
Returns (conv_snapshot, rec_snapshot). Each tensor is deep-copied so that subsequent SSM mutations during TTT adaptation do not corrupt the snapshot.
Sourcepub fn restore_ssm_states(
&self,
conv_snapshot: Vec<Option<Tensor>>,
rec_snapshot: Vec<Option<Tensor>>,
)
pub fn restore_ssm_states( &self, conv_snapshot: Vec<Option<Tensor>>, rec_snapshot: Vec<Option<Tensor>>, )
Restore SSM states from a snapshot produced by snapshot_ssm_states.
Sourcepub fn has_mtp(&self) -> bool
pub fn has_mtp(&self) -> bool
Whether a usable MTP draft head was loaded (dense checkpoints only).
Like ModelOperations::forward_with_cache but also returns the
pre-final-norm hidden states for ALL forwarded positions — the MTP head
consumes them as its h input. With last_logits_only the LM head runs
over the last position only (the #201 prefill optimization); otherwise
every position’s logits are returned (the 2-token verify step needs both).
Returns (logits [1, seq|1, vocab], hidden [1, seq, hidden]).
Sourcepub fn mtp_forward(
&self,
token_ids: &Tensor,
hiddens: &Tensor,
base: usize,
) -> Result<Tensor, Error>
pub fn mtp_forward( &self, token_ids: &Tensor, hiddens: &Tensor, base: usize, ) -> Result<Tensor, Error>
Run the MTP draft head over seq consecutive slots.
Slot i pairs hiddens[.., i, ..] (the main model’s pre-norm hidden
after some token x_t) with embed(token_ids[.., i]) (the next token
x_{t+1}) at absolute RoPE/KV position base + i, predicting x_{t+2}.
Every slot’s KV is written to the MTP cache slot (keeping it contiguous
with prior drafts); only the LAST slot’s logits are returned — that is
the draft for the position after the last consumed token.
Returns logits [1, 1, vocab].
Trait Implementations§
Source§impl ModelOperations for Qwen3_5Model
impl ModelOperations for Qwen3_5Model
Source§fn architecture(&self) -> ModelArchitecture
fn architecture(&self) -> ModelArchitecture
Source§fn config(&self) -> &dyn ArchitectureConfig
fn config(&self) -> &dyn ArchitectureConfig
Source§fn decision_output_embedding_rows(&self, ids: &[i64]) -> Result<Tensor, Error>
fn decision_output_embedding_rows(&self, ids: &[i64]) -> Result<Tensor, Error>
Source§fn forward(
&self,
input: &Tensor,
_past_kv: Option<&Tensor>,
) -> Result<Tensor, Error>
fn forward( &self, input: &Tensor, _past_kv: Option<&Tensor>, ) -> Result<Tensor, Error>
Source§fn forward_with_cache(
&self,
input: &Tensor,
start_pos: usize,
) -> Result<Tensor, Error>
fn forward_with_cache( &self, input: &Tensor, start_pos: usize, ) -> Result<Tensor, Error>
Source§fn forward_from_embeddings(
&self,
embeddings: &Tensor,
start_pos: usize,
) -> Result<Tensor, Error>
fn forward_from_embeddings( &self, embeddings: &Tensor, start_pos: usize, ) -> Result<Tensor, Error>
Source§fn forward_with_cache_and_delta(
&self,
input: &Tensor,
start_pos: usize,
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
fn forward_with_cache_and_delta( &self, input: &Tensor, start_pos: usize, delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Source§fn embed_tokens(&self, input_ids: &Tensor) -> Result<Tensor, Error>
fn embed_tokens(&self, input_ids: &Tensor) -> Result<Tensor, Error>
Source§fn apply_final_norm(&self, hidden_states: &Tensor) -> Result<Tensor, Error>
fn apply_final_norm(&self, hidden_states: &Tensor) -> Result<Tensor, Error>
Source§fn lm_head(&self, hidden_states: &Tensor) -> Result<Tensor, Error>
fn lm_head(&self, hidden_states: &Tensor) -> Result<Tensor, Error>
Source§fn num_layers(&self) -> usize
fn num_layers(&self) -> usize
Source§fn forward_layers(
&self,
hidden: &Tensor,
range: Range<usize>,
start_pos: usize,
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
fn forward_layers( &self, hidden: &Tensor, range: Range<usize>, start_pos: usize, delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Source§fn forward_layers_train(
&self,
hidden: &Tensor,
range: Range<usize>,
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
fn forward_layers_train( &self, hidden: &Tensor, range: Range<usize>, delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Self::forward_layers — the cross-device
autograd primitive for TTT-on-split (#316, M-TRAIN-COUPLING). Read moreSource§fn decode_layer(
&self,
layer_idx: usize,
hidden_states: &Tensor,
_attention_mask: Option<&Tensor>,
position_ids: Option<&Tensor>,
_past_kv: Option<&LayerKVCache>,
) -> Result<(Tensor, Option<(Tensor, Tensor)>), Error>
fn decode_layer( &self, layer_idx: usize, hidden_states: &Tensor, _attention_mask: Option<&Tensor>, position_ids: Option<&Tensor>, _past_kv: Option<&LayerKVCache>, ) -> Result<(Tensor, Option<(Tensor, Tensor)>), Error>
Source§fn decode_layer_with_delta(
&self,
layer_idx: usize,
hidden_states: &Tensor,
_attention_mask: Option<&Tensor>,
position_ids: Option<&Tensor>,
_past_kv: Option<&LayerKVCache>,
delta: &TenantDelta,
) -> Result<(Tensor, Option<(Tensor, Tensor)>), Error>
fn decode_layer_with_delta( &self, layer_idx: usize, hidden_states: &Tensor, _attention_mask: Option<&Tensor>, position_ids: Option<&Tensor>, _past_kv: Option<&LayerKVCache>, delta: &TenantDelta, ) -> Result<(Tensor, Option<(Tensor, Tensor)>), Error>
Source§fn clear_kv_cache(&self)
fn clear_kv_cache(&self)
Source§fn set_kv_cache(&mut self, cache: Arc<Mutex<RawMutex, KVCacheManager>>)
fn set_kv_cache(&mut self, cache: Arc<Mutex<RawMutex, KVCacheManager>>)
Source§fn get_kv_cache(&self) -> Option<Arc<Mutex<RawMutex, KVCacheManager>>>
fn get_kv_cache(&self) -> Option<Arc<Mutex<RawMutex, KVCacheManager>>>
Source§fn take_kv_cache(&mut self) -> Option<Arc<Mutex<RawMutex, KVCacheManager>>>
fn take_kv_cache(&mut self) -> Option<Arc<Mutex<RawMutex, KVCacheManager>>>
Source§fn reshape_for_attention(
&self,
tensor: &Tensor,
_is_key_value: bool,
) -> Result<Tensor, Error>
fn reshape_for_attention( &self, tensor: &Tensor, _is_key_value: bool, ) -> Result<Tensor, Error>
Source§fn apply_rope(
&self,
tensor: &Tensor,
position_ids: &Tensor,
) -> Result<Tensor, Error>
fn apply_rope( &self, tensor: &Tensor, position_ids: &Tensor, ) -> Result<Tensor, Error>
Source§fn normalize(&self, tensor: &Tensor) -> Result<Tensor, Error>
fn normalize(&self, tensor: &Tensor) -> Result<Tensor, Error>
Source§fn get_attention_mask(
&self,
seq_len: usize,
_past_kv_len: usize,
) -> Result<Tensor, Error>
fn get_attention_mask( &self, seq_len: usize, _past_kv_len: usize, ) -> Result<Tensor, Error>
Source§fn is_multimodal(&self) -> bool
fn is_multimodal(&self) -> bool
Source§fn prepare_multimodal_inputs(
&self,
input_ids: &Tensor,
pixel_values: Option<&Tensor>,
images_seq_mask: Option<&Tensor>,
images_emb_mask: Option<&Tensor>,
) -> Result<Tensor, Error>
fn prepare_multimodal_inputs( &self, input_ids: &Tensor, pixel_values: Option<&Tensor>, images_seq_mask: Option<&Tensor>, images_emb_mask: Option<&Tensor>, ) -> Result<Tensor, Error>
Source§fn encode_vision(&self, pixel_values: &Tensor) -> Result<Tensor, Error>
fn encode_vision(&self, pixel_values: &Tensor) -> Result<Tensor, Error>
Source§fn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
Source§fn get_tokenizer_config(&self) -> Box<dyn TokenizerConfig>
fn get_tokenizer_config(&self) -> Box<dyn TokenizerConfig>
Source§fn var_store_mut(&mut self) -> Option<&mut VarStore>
fn var_store_mut(&mut self) -> Option<&mut VarStore>
Source§fn forward_batched(
&self,
_sequences: &mut [(Vec<i64>, usize, Arc<Mutex<RawMutex, KVCacheManager>>)],
_delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
fn forward_batched( &self, _sequences: &mut [(Vec<i64>, usize, Arc<Mutex<RawMutex, KVCacheManager>>)], _delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Source§fn kv_cache_memory_usage(&self) -> usize
fn kv_cache_memory_usage(&self) -> usize
impl Send for Qwen3_5Model
impl Sync for Qwen3_5Model
Auto Trait Implementations§
impl Freeze for Qwen3_5Model
impl !RefUnwindSafe for Qwen3_5Model
impl Unpin for Qwen3_5Model
impl UnsafeUnpin for Qwen3_5Model
impl !UnwindSafe for Qwen3_5Model
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request