Expand description
Weight provider for streaming large models
This module provides an abstraction for loading model weights on-demand, allowing models larger than system memory to be loaded.
Structs§
- Memory
Weight Provider - Simple in-memory weight provider (current behavior) Wraps tensors in Arc<Mutex<>> for thread safety
- Streaming
Weight Provider - Streaming weight provider that loads shards on demand
Traits§
- Weight
Provider - Trait for providing weights to models