pub struct ModelFactory;Expand description
Factory for creating models with proper configuration management
Implementations§
Source§impl ModelFactory
impl ModelFactory
Sourcepub async fn detect_model_dtype(model_path: &Path) -> Result<DType>
pub async fn detect_model_dtype(model_path: &Path) -> Result<DType>
Detect the dtype of a model by examining its tensors
Sourcepub async fn create(
model_path: &Path,
device: &Device,
dtype: DType,
max_context: Option<usize>,
kv_quant_type: KVQuantType,
device_pool: Option<&DevicePool>,
fp8_dequant_load: bool,
speculative_decoding: bool,
) -> Result<Box<dyn ModelOperations>>
pub async fn create( model_path: &Path, device: &Device, dtype: DType, max_context: Option<usize>, kv_quant_type: KVQuantType, device_pool: Option<&DevicePool>, fp8_dequant_load: bool, speculative_decoding: bool, ) -> Result<Box<dyn ModelOperations>>
Create a model from a directory containing weights and optionally config.json This is the ONLY way models should be created to ensure consistency
§Multi-device pipeline (#314 wiring, #310 epic)
Pass device_pool = Some(pool) with a pool of >1 device to build the model
as a single multi-device pipeline stage: decoder layers are spread across
the pool’s devices via [LayerDeviceMap::even_split], and the model’s
forward_layers performs the lone cross-device copy at each stage boundary.
device must be the pool’s primary (first) device — non-layer weights
(embeddings, final norm, lm_head) live there. When device_pool is None
or holds a single device, construction is unchanged (whole model on
device). At runtime this depends on #405 (from_weights device-placement
fix) to actually place per-layer tensors on non-primary devices.
speculative_decoding is the resolved runtime policy (including TOML).
When false, KV is quantized, or config/checkpoint metadata declares an
unsupported MTP layout, draft weights skip tensor/device allocation.
Sourcepub async fn create_stage(
model_path: &Path,
device: &Device,
dtype: DType,
max_context: Option<usize>,
kv_quant_type: KVQuantType,
device_pool: Option<&DevicePool>,
request: ModelStageRequest,
fp8_dequant_load: bool,
) -> Result<Box<dyn ModelOperations>>
pub async fn create_stage( model_path: &Path, device: &Device, dtype: DType, max_context: Option<usize>, kv_quant_type: KVQuantType, device_pool: Option<&DevicePool>, request: ModelStageRequest, fp8_dequant_load: bool, ) -> Result<Box<dyn ModelOperations>>
Create one Llama-family pipeline stage from an authoritative HF index.
Unlike Self::create, stage mode never falls back to a single-file or
glob-discovered whole-model load. It requires
model.safetensors.index.json, resolves the exact tensor names for the
requested global layer range and its first/last role, opens only shards
containing those names, and constructs no tensor outside that set.
Llama, Qwen, and Mistral checkpoints use the production-wired
LlamaModel stage path. Qwen3.5 remains deliberately out of scope.
Sourcepub async fn create_with_fs(
model_path: &Path,
device: &Device,
dtype: DType,
max_context: Option<usize>,
kv_quant_type: KVQuantType,
fs: &WorktreeClient,
device_pool: Option<&DevicePool>,
fp8_dequant_load: bool,
speculative_decoding: bool,
) -> Result<Box<dyn ModelOperations>>
pub async fn create_with_fs( model_path: &Path, device: &Device, dtype: DType, max_context: Option<usize>, kv_quant_type: KVQuantType, fs: &WorktreeClient, device_pool: Option<&DevicePool>, fp8_dequant_load: bool, speculative_decoding: bool, ) -> Result<Box<dyn ModelOperations>>
Create a model using FsOps for weight loading.
Uses FsOps::read_file() instead of direct filesystem access.
The model_path is still needed for ModelConfig and architecture detection
(which parse config.json), but weight data is read through FsOps.
Explicit policy and whole-checkpoint layout preflight filter MTP tensors
before materialization here too; policy is not re-read from the environment.
Sourcepub async fn detect_model_dtype_fs(fs: &WorktreeClient) -> Result<DType>
pub async fn detect_model_dtype_fs(fs: &WorktreeClient) -> Result<DType>
Detect model dtype using FsOps for file reading.
Auto Trait Implementations§
impl Freeze for ModelFactory
impl RefUnwindSafe for ModelFactory
impl Send for ModelFactory
impl Sync for ModelFactory
impl Unpin for ModelFactory
impl UnsafeUnpin for ModelFactory
impl UnwindSafe for ModelFactory
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more