Skip to main content

Module weight_provider

Module weight_provider 

Source
Expand description

Weight provider for streaming large models

This module provides an abstraction for loading model weights on-demand, allowing models larger than system memory to be loaded.

Structs§

MemoryWeightProvider
Simple in-memory weight provider (current behavior) Wraps tensors in Arc<Mutex<>> for thread safety
StreamingWeightProvider
Streaming weight provider that loads shards on demand

Traits§

WeightProvider
Trait for providing weights to models