pub struct TorchEngine {
pub active_lora: Arc<Mutex<RawMutex, Option<String>>>,
/* private fields */
}Expand description
PyTorch inference engine using tch-rs
Thread-safe implementation using proper synchronization primitives. All mutable state is protected by mutexes with poisoning recovery.
Fields§
§active_lora: Arc<Mutex<RawMutex, Option<String>>>Active LoRA adapter name - thread safe with mutex
Implementations§
Source§impl TorchEngine
Helper functions for tensor operations
impl TorchEngine
Helper functions for tensor operations
Sourcepub fn set_seed(&self, seed: u64)
pub fn set_seed(&self, seed: u64)
Set the random seed for deterministic generation Useful for debugging - enables reproducible token sequences
Sourcepub fn get_vocab_size(&self) -> usize
pub fn get_vocab_size(&self) -> usize
Get cached vocabulary size without locking
Sourcepub fn device_pool(&self) -> Option<&DevicePool>
pub fn device_pool(&self) -> Option<&DevicePool>
The resolved multi-device pool, if this engine was built with >1 device.
Returns None for single-device engines (the common case, including the
legacy single-gpu_device_id path and CPU). When Some and non-single,
the loaded model is built as a single pipeline stage with a layer→device
map derived from this pool (#314 wiring). The pool’s primary equals
Self::device.
Sourcepub fn is_multi_device(&self) -> bool
pub fn is_multi_device(&self) -> bool
Whether this engine is configured to build a multi-device pipeline.
True exactly when Self::device_pool is Some and holds >1 device.
The model is constructed with LayerDeviceMap::even_split across the
pool; single-device engines return false and build the whole model on
Self::device.
Sourcepub fn initialize_kv_registry(
&mut self,
num_layers: usize,
max_seq_len: usize,
quant_type: KVQuantType,
memory_budget: Option<usize>,
)
pub fn initialize_kv_registry( &mut self, num_layers: usize, max_seq_len: usize, quant_type: KVQuantType, memory_budget: Option<usize>, )
Initialize the KV cache registry for session-based cache isolation.
This should be called after loading a model when the configuration is known. The registry will manage separate KV caches for different sessions/requests.
Sourcepub fn kv_compat_fingerprint_for(
&self,
adapter_generation: u64,
) -> Option<KvCompatFingerprint>
pub fn kv_compat_fingerprint_for( &self, adapter_generation: u64, ) -> Option<KvCompatFingerprint>
Expected KV-cache compatibility fingerprint for the loaded model under the
given adapter/delta generation, or None when no authoritative identity is
available.
The effective identity folds in the base-weight content digest, the full
effective architecture/runtime config, and the live adapter_generation
actually applied to the request. Returns None when no descriptor was
built (the base-weight digest could not be computed) or when the descriptor
is not yet authoritatively complete (tokenizer identity not yet folded in,
or tokenizer hashing failed). The session-reuse path treats None as
fail-closed: it discards a cache rather than reusing it under a
missing or uncertain identity — it never treats absence as “no gating”.
Sourcepub fn compute_kv_budget(&self) -> Option<usize>
pub fn compute_kv_budget(&self) -> Option<usize>
Compute a KV cache memory budget based on model size and available GPU memory.
Uses HYPRSTREAM_KV_BUDGET_MB env var if set, otherwise estimates:
GPU total (from common card sizes) minus model weights, times 0.7.
Returns None on CPU (no budget needed).
Sourcepub fn kv_registry(&self) -> Option<Arc<KVCacheRegistry>>
pub fn kv_registry(&self) -> Option<Arc<KVCacheRegistry>>
Get the KV cache registry (for external access)
Sourcepub fn set_session(&self, owner: CacheOwner) -> Result<(), Error>
pub fn set_session(&self, owner: CacheOwner) -> Result<(), Error>
Set the active session for generation.
This sets which KV cache will be used for the next generation call.
Use CacheOwner::Session(session_id) for conversational sessions that
should preserve context across multiple requests.
§Example
// For a conversational session (preserves context)
engine.set_session(CacheOwner::Session("conversation-123".into()));
// For a stateless request (cache discarded after)
engine.set_session(CacheOwner::new_stateless());Sourcepub fn get_session(&self) -> Option<CacheOwner>
pub fn get_session(&self) -> Option<CacheOwner>
Get the current active session owner
Sourcepub fn clear_session(&self) -> Result<(), Error>
pub fn clear_session(&self) -> Result<(), Error>
Clear the active session (subsequent generations will use the model’s internal cache)
Sourcepub fn release_session(&self, owner: &CacheOwner) -> Result<(), Error>
pub fn release_session(&self, owner: &CacheOwner) -> Result<(), Error>
Release a session’s KV cache (for cleanup after conversation ends)
Sourcepub fn session_cache_memory_usage(&self) -> usize
pub fn session_cache_memory_usage(&self) -> usize
Get total memory usage across all session caches
Sourcepub fn active_session_count(&self) -> usize
pub fn active_session_count(&self) -> usize
Get number of active session caches
Sourcepub fn evict_session_caches(&self)
pub fn evict_session_caches(&self)
Evict LRU session caches to stay within memory budget
Sourcepub fn swap_session_cache(
&self,
tenant_id: Option<String>,
weight_epoch: u64,
) -> bool
pub fn swap_session_cache( &self, tenant_id: Option<String>, weight_epoch: u64, ) -> bool
Swap the active session’s KV cache into the model.
If an active session owner is set and the registry is initialized,
retrieves (or creates) the session’s cache from the registry and
installs it in the model via set_kv_cache(). Returns true if
a session cache was swapped in.
tenant_id is the effective tenant whose delta contributed to this
request’s weights (None when no tenant delta contributes), and
weight_epoch is that delta’s current weight epoch (ignored when
tenant_id is None). The pair is registered against the owner cache
before the cache is installed in the model, so the cache is tracked
for invalidation (and cleared if its previously-computed KV is stale
w.r.t. a different tenant or an older weight epoch) before generation
can read a single token from it.
Atomicity invariant (F2): the register→install sequence below is
not guarded by a lock spanning both operations. It is race-free today
because InferenceService runs on a single-threaded current_thread
Tokio runtime inside a LocalSet, and this function is await-free — no
other request task can interleave between register_delta_dependency
and set_kv_cache, so the pair executes atomically with respect to
other requests. If InferenceService ever moves to a multi-threaded
runtime (or an .await is introduced here), this invariant breaks and a
spanning lock or an epoch handshake would be required.
Sourcepub fn model_memory_usage(&self) -> usize
pub fn model_memory_usage(&self) -> usize
Estimate model memory usage in bytes.
Sums the byte sizes of all parameters in the VarStore.
Sourcepub fn new(config: RuntimeConfig) -> Result<TorchEngine, Error>
pub fn new(config: RuntimeConfig) -> Result<TorchEngine, Error>
Create new PyTorch engine
Sourcepub async fn new_async(config: RuntimeConfig) -> Result<TorchEngine, Error>
pub async fn new_async(config: RuntimeConfig) -> Result<TorchEngine, Error>
Create new PyTorch engine (async version)
Sourcepub fn get_tensor(&self, name: &str) -> Option<Tensor>
pub fn get_tensor(&self, name: &str) -> Option<Tensor>
Get tensor from VarStore by name (for inference) - thread safe
Sourcepub fn list_tensor_names(&self) -> Vec<String>
pub fn list_tensor_names(&self) -> Vec<String>
List all available tensor names in VarStore - thread safe
Sourcepub fn has_varstore(&self) -> bool
pub fn has_varstore(&self) -> bool
Check if model is loaded via VarStore - thread safe
Sourcepub fn is_eos_token(&self, token_id: usize) -> bool
pub fn is_eos_token(&self, token_id: usize) -> bool
Check if a token ID is the EOS token (lock-free using cached ID)
Sourcepub fn get_tokenizer(&self) -> Result<Tokenizer, Error>
pub fn get_tokenizer(&self) -> Result<Tokenizer, Error>
Get the tokenizer for streaming decoding - CoW makes this cheap!
Returns a cloned tokenizer (cheap due to copy-on-write)
Sourcepub fn forward(&self, input_ids: &[i64]) -> Result<Tensor, Error>
pub fn forward(&self, input_ids: &[i64]) -> Result<Tensor, Error>
Run inference on the model (supports both TorchScript and VarStore models) - thread safe
Sourcepub fn forward_cached(
&self,
input_ids: &[i64],
start_pos: usize,
) -> Result<Tensor, Error>
pub fn forward_cached( &self, input_ids: &[i64], start_pos: usize, ) -> Result<Tensor, Error>
Run optimized inference with KV caching - only process new tokens
Sourcepub fn forward_with_delta_cached(
&self,
input_ids: &[i64],
start_pos: usize,
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
pub fn forward_with_delta_cached( &self, input_ids: &[i64], start_pos: usize, delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Run inference with KV caching and optional delta injection
Same as forward_cached() but calls forward_with_cache_and_delta() on the model,
injecting per-tenant LoRA corrections at each attention layer.
Sourcepub fn forward_batched_step(
&self,
sequences: &mut [(Vec<i64>, usize, Arc<Mutex<RawMutex, KVCacheManager>>)],
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
pub fn forward_batched_step( &self, sequences: &mut [(Vec<i64>, usize, Arc<Mutex<RawMutex, KVCacheManager>>)], delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Batched ragged decode step over several same-delta sequences (#329).
Each entry is (new_token_ids, start_pos, per-sequence KVCacheManager).
Returns last-token logits per row [B, vocab]. Delegates to the model’s
forward_batched (Llama-only; other architectures return an error so the
caller falls back to per-stream forward_with_delta_cached). Tensors stay
on this (bridge) thread; only the Send token/logits metadata crosses.
Sourcepub fn forward_with_delta_full(
&self,
input_ids: &[i64],
delta: Option<&TenantDelta>,
) -> Result<Tensor, Error>
pub fn forward_with_delta_full( &self, input_ids: &[i64], delta: Option<&TenantDelta>, ) -> Result<Tensor, Error>
Run full forward pass with optional delta injection (no KV cache)
Sourcepub fn clear_kv_cache(&self)
pub fn clear_kv_cache(&self)
Clear KV cache before new generation to prevent context pollution
Sourcepub fn snapshot_ssm_states(
&self,
) -> Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>
pub fn snapshot_ssm_states( &self, ) -> Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>
Snapshot Qwen3.5 SSM (conv/rec) states for TTT adaptation.
Returns Some((conv, rec)) if the loaded model is Qwen3.5; None otherwise.
Each Tensor is deep-copied via .copy() (C4 fix) so the snapshot is
independent of subsequent forward-pass mutations during the TTT loop.
Sourcepub fn restore_ssm_states(
&self,
snapshot: Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>,
)
pub fn restore_ssm_states( &self, snapshot: Option<(Vec<Option<Tensor>>, Vec<Option<Tensor>>)>, )
Restore Qwen3.5 SSM states from a snapshot produced by snapshot_ssm_states.
No-op if the loaded model is not Qwen3.5 or snapshot is None.
Sourcepub fn speculative_supported(&self) -> bool
pub fn speculative_supported(&self) -> bool
Whether the loaded model can self-speculate: a Qwen3.5 with a loaded (dense) MTP draft head. Pipeline/multi-device stage loads never own the MTP head, so this is false there too.
Sourcepub fn forward_speculative(
&self,
input_ids: &[i64],
start_pos: usize,
last_logits_only: bool,
) -> Result<(Tensor, Tensor), Error>
pub fn forward_speculative( &self, input_ids: &[i64], start_pos: usize, last_logits_only: bool, ) -> Result<(Tensor, Tensor), Error>
Speculative prefill/verify forward: returns per-position logits plus the pre-final-norm hidden states the MTP head consumes.
Never delta-adapted: speculation is disabled for tenant-delta streams at stream construction, so this always runs the base model.
Returns (logits [1, seq|1, vocab], hidden [1, seq, hidden_size]).
Sourcepub fn forward_mtp_draft(
&self,
token_ids: &[i64],
hiddens: &Tensor,
base: usize,
) -> Result<Tensor, Error>
pub fn forward_mtp_draft( &self, token_ids: &[i64], hiddens: &Tensor, base: usize, ) -> Result<Tensor, Error>
MTP draft forward over token_ids.len() consecutive slots. Slot i
pairs hiddens[.., i, ..] with embed(token_ids[i]) at absolute
position base + i. Returns the last slot’s logits [1, 1, vocab].
Source§impl TorchEngine
impl TorchEngine
Sourcepub fn enable_training(&mut self, _learning_rate: f64) -> Result<(), Error>
pub fn enable_training(&mut self, _learning_rate: f64) -> Result<(), Error>
Enable training mode for pre-training
Sourcepub fn manual_sgd_step(&mut self, _learning_rate: f32) -> Result<(), Error>
pub fn manual_sgd_step(&mut self, _learning_rate: f32) -> Result<(), Error>
Perform a manual SGD step without optimizer
Sourcepub fn forward_training(
&self,
input_ids: &Tensor,
track_gradients: bool,
) -> Result<Tensor, Error>
pub fn forward_training( &self, input_ids: &Tensor, track_gradients: bool, ) -> Result<Tensor, Error>
Forward pass for training (with gradient tracking)
Sourcepub fn compute_loss_and_backward(
&self,
input_ids: &Tensor,
labels: &Tensor,
) -> Result<f64, Error>
pub fn compute_loss_and_backward( &self, input_ids: &Tensor, labels: &Tensor, ) -> Result<f64, Error>
Compute loss and backward (without optimizer step)
Sourcepub fn disable_training(&mut self) -> Result<(), Error>
pub fn disable_training(&mut self) -> Result<(), Error>
Disable training mode
Sourcepub fn is_training_enabled(&self) -> bool
pub fn is_training_enabled(&self) -> bool
Check if training is enabled
Source§impl TorchEngine
impl TorchEngine
Sourcepub fn is_persistent_model_ready(&self) -> bool
pub fn is_persistent_model_ready(&self) -> bool
Check if persistent model is initialized - thread safe
Sourcepub fn generate(
&self,
request: GenerationRequest,
) -> Result<TextStream<'_>, Error>
pub fn generate( &self, request: GenerationRequest, ) -> Result<TextStream<'_>, Error>
Generate text as a stream of decoded UTF-8 text chunks
Returns a Stream that yields Result<String> with properly decoded text.
The stream automatically handles:
- Multi-byte UTF-8 sequences (emojis, CJK characters, etc.)
- EOS and stop token detection
- Max tokens limit
- Special token filtering
Dropping the stream stops generation automatically (no manual cancellation needed).
§Example
use futures::StreamExt;
use hyprstream_rpc_std::inference_client::GenerationRequest;
let request = GenerationRequest::default();
let mut stream = engine.generate(request)?;
while let Some(text_chunk) = stream.next().await {
print!("{}", text_chunk?);
}
let stats = stream.stats();
println!("Generated {} tokens", stats.tokens_generated);Sourcepub fn generate_with_delta(
&self,
request: GenerationRequest,
delta: Option<Arc<Mutex<RawMutex, TenantDelta>>>,
tenant_id: Option<String>,
weight_epoch: u64,
adapter_generation: u64,
) -> Result<TextStream<'_>, Error>
pub fn generate_with_delta( &self, request: GenerationRequest, delta: Option<Arc<Mutex<RawMutex, TenantDelta>>>, tenant_id: Option<String>, weight_epoch: u64, adapter_generation: u64, ) -> Result<TextStream<'_>, Error>
Generate with optional per-tenant delta for delta-aware inference
When delta is Some, LoRA corrections are injected at each attention layer
during generation. The delta is locked per-token (not held across the entire
generation) to allow concurrent training updates.
adapter_generation is the live LoRA/delta generation actually applied to
this request; it is folded into the KV-compat expected identity so a
re-adapted model is detected as incompatible and its stale KV rejected.
tenant_id is the effective tenant whose delta contributed to the weights
(Some only when a per-tenant delta is in play, None otherwise), and
weight_epoch is that delta’s current weight epoch. The pair is threaded
to the session-cache swap so the resulting KV is registered for
tenant-delta invalidation and cleared if the weights changed under a
different tenant or an older weight epoch (same-tenant in-place
re-adaptation). The caller must derive the tenant id from the verified
request subject, not invent a new identity source.
Sourcepub async fn generate_with_delta_params(
&self,
request: GenerationRequest,
delta: Option<Arc<Mutex<RawMutex, TenantDelta>>>,
tenant_id: Option<String>,
weight_epoch: u64,
adapter_generation: u64,
) -> Result<GenerationResult, Error>
pub async fn generate_with_delta_params( &self, request: GenerationRequest, delta: Option<Arc<Mutex<RawMutex, TenantDelta>>>, tenant_id: Option<String>, weight_epoch: u64, adapter_generation: u64, ) -> Result<GenerationResult, Error>
Non-streaming generation with optional delta (convenience wrapper)
Sourcepub fn forward_with_delta(
&self,
input_ids: &Tensor,
delta: &TenantDelta,
) -> Result<Tensor, Error>
pub fn forward_with_delta( &self, input_ids: &Tensor, delta: &TenantDelta, ) -> Result<Tensor, Error>
Forward pass with per-layer delta injection for TTT training.
Uses decode_layer_with_delta() to inject the tenant delta’s A/B matrices
after q_proj/v_proj projections inside each attention layer. This creates a
differentiable path from the loss back to the delta parameters, enabling
gradient-based training via loss.backward().
§Arguments
input_ids- Token IDs [1, seq_len]delta- Tenant’s LoRA delta with trainable A/B matrices
§Returns
Logits tensor [1, seq_len, vocab_size] with gradient path to delta parameters
Sourcepub fn effective_context_limit(&self) -> Result<usize, Error>
pub fn effective_context_limit(&self) -> Result<usize, Error>
Authoritative context capacity of the loaded model. ModelInfo is populated from the checkpoint configuration and optional max_context override during loading; RuntimeConfig.context_length is only a constructor default. Serving encoders and native decision admission must use this same limit.
Sourcepub fn with_decision_backbone<T>(
&self,
token_ids: &[i64],
head: impl FnOnce(&Tensor, &dyn Fn(&[i64]) -> Result<Tensor, Error>) -> Result<T, Error>,
) -> Result<T, Error>
pub fn with_decision_backbone<T>( &self, token_ids: &[i64], head: impl FnOnce(&Tensor, &dyn Fn(&[i64]) -> Result<Tensor, Error>) -> Result<T, Error>, ) -> Result<T, Error>
Execute a decision head against fresh full-sequence hidden states while holding the backbone lock. The supplied gather reads original LM-head rows without allocating token-by-vocabulary logits. No cache is reused.
Sourcepub fn with_decision_backbone_checked<T>(
&self,
token_ids: &[i64],
check: &dyn Fn() -> Result<(), Error>,
head: impl FnOnce(&Tensor, &dyn Fn(&[i64]) -> Result<Tensor, Error>) -> Result<T, Error>,
) -> Result<T, Error>
pub fn with_decision_backbone_checked<T>( &self, token_ids: &[i64], check: &dyn Fn() -> Result<(), Error>, head: impl FnOnce(&Tensor, &dyn Fn(&[i64]) -> Result<Tensor, Error>) -> Result<T, Error>, ) -> Result<T, Error>
Cooperative cancellation/deadline variant. The checker runs before admission, between decoder layers, and around head evaluation. An active libtorch kernel completes before the next check; dropping a future alone is not a compute-cancellation mechanism.
Sourcepub fn get_num_layers(&self) -> Result<usize, Error>
pub fn get_num_layers(&self) -> Result<usize, Error>
Get the number of layers in the loaded model.
Sourcepub fn get_lora_module_dims(
&self,
) -> Result<HashMap<String, (usize, usize)>, Error>
pub fn get_lora_module_dims( &self, ) -> Result<HashMap<String, (usize, usize)>, Error>
Get module dimensions for LoRA target modules from the loaded model.
Returns a map of module_name -> (in_features, out_features) for all supported LoRA target modules. Used by DeltaPool to initialize per-tenant LoRA deltas with correct dimensions.
Supported modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Sourcepub fn get_per_layer_lora_dims(
&self,
) -> Option<HashMap<usize, HashMap<String, (usize, usize)>>>
pub fn get_per_layer_lora_dims( &self, ) -> Option<HashMap<usize, HashMap<String, (usize, usize)>>>
Get per-layer LoRA module dimensions for hybrid architectures (e.g., Qwen3.5).
Returns None for uniform architectures (Llama, Gemma, etc.) where all layers
have identical module dims. Returns per-layer overrides for Qwen3.5 where GDN
layers have different o_proj input dims than full-attention layers.
Sourcepub fn create_lora(&mut self, config: TenantDeltaConfig) -> Result<(), Error>
pub fn create_lora(&mut self, config: TenantDeltaConfig) -> Result<(), Error>
Validate LoRA configuration against model dimensions.
Checks that the requested target modules are valid for this model. The actual LoRA delta creation is handled by DeltaPool/TenantDelta.
Sourcepub fn extract_embedding(&self, text: &str) -> Result<Vec<f32>, Error>
pub fn extract_embedding(&self, text: &str) -> Result<Vec<f32>, Error>
Extract embedding from text using the model’s hidden states.
This method tokenizes the input text, runs a forward pass through the model, and extracts the final hidden states. The embedding is computed by mean-pooling the hidden states over the sequence dimension.
§Arguments
text- Input text to embed
§Returns
- Vec
- The embedding vector (dimension = model’s hidden_size)
§Example
let embedding = engine.extract_embedding("What is machine learning?")?;
println!("Embedding dimension: {}", embedding.len());Sourcepub fn extract_embedding_from_tokens(
&self,
token_ids: &[i64],
) -> Result<Vec<f32>, Error>
pub fn extract_embedding_from_tokens( &self, token_ids: &[i64], ) -> Result<Vec<f32>, Error>
Extract embedding from pre-tokenized input.
This is a lower-level method for when tokens are already available (e.g., from a previous generation). Uses mean pooling over the hidden states.
§Arguments
token_ids- Pre-tokenized input IDs
§Returns
- Vec
- The embedding vector (dimension = model’s hidden_size)
Sourcepub fn embed_images(&self, images: &[Vec<u8>]) -> Result<Vec<Vec<f32>>, Error>
pub fn embed_images(&self, images: &[Vec<u8>]) -> Result<Vec<Vec<f32>>, Error>
Compute vision embeddings for one or more images.
Preprocesses raw image bytes and runs them through the model’s vision encoder (e.g. SigLIP). Returns one embedding vector per image.
§Arguments
images- Raw image bytes (PNG/JPEG/RGB) per image
§Returns
- Vec<Vec
> - Embedding vectors, one per input image
Sourcepub fn release_device_resources(&mut self)
pub fn release_device_resources(&mut self)
Explicitly drop every CUDA/device tensor the engine owns or has cached on this thread, now — while the CUDA context is still alive.
§Why this exists (#429)
tch/libtorch defers CUDA context teardown to an atexit handler that
runs during process exit. But two classes of device tensor outlive the
engine struct and only drop later:
- Thread-local RoPE caches (
llama::ROPE_CACHE,qwen3_5::ROPE_CACHE) — sin/cos tables cached on the compute device. They are TLS, so they are not touched byDropand survive until the owning thread exits. - The inference service thread itself — it is spawned detached in
LocalInferenceService::start, so its TLS destructors (and thus the RoPE cache drop) can fire during process-exit cleanup, after the CUDA context is gone →c10::Error: invalid device pointer(SIGABRT).
The engine-only GPU smoke (e6_gpu_engine_is_single_device_on_one_gpu_host)
never loads weights nor runs a forward pass, so it populates no RoPE
cache and exits cleanly; the streaming test does, which is why only it
aborts.
Calling this on the service thread at Shutdown receipt — before
replying to the client — drops the RoPE tables and the model weights
synchronously while the caller is still blocked on the reply (process
indubitably alive, context valid). After it returns only empty shells
remain for any late TLS/static destruction, so there is nothing left to
free against a dead context.
§Contract
This is a terminal teardown: the engine is unusable afterward
(model_info reports unloaded, generation will error). It must only be
called once, at shutdown. Unlike Drop, it deliberately does clear
shared Arc<Mutex<Option<T>>> contents — that is safe precisely because
the engine is being retired, not because the contents are unused. (The
production inference path holds a single engine instance per service; a
cached clone would see None and fail closed with “No model loaded”,
which is the correct behavior for a retired engine.)
Trait Implementations§
Source§impl Clone for TorchEngine
impl Clone for TorchEngine
Source§fn clone(&self) -> TorchEngine
fn clone(&self) -> TorchEngine
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreSource§impl Drop for TorchEngine
impl Drop for TorchEngine
Source§impl RuntimeEngine for TorchEngine
impl RuntimeEngine for TorchEngine
Source§fn load_model<'life0, 'life1, 'async_trait>(
&'life0 mut self,
path: &'life1 Path,
) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
TorchEngine: 'async_trait,
fn load_model<'life0, 'life1, 'async_trait>(
&'life0 mut self,
path: &'life1 Path,
) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
TorchEngine: 'async_trait,
Source§fn generate<'life0, 'life1, 'async_trait>(
&'life0 self,
prompt: &'life1 str,
max_tokens: usize,
) -> Pin<Box<dyn Future<Output = Result<String, Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
TorchEngine: 'async_trait,
fn generate<'life0, 'life1, 'async_trait>(
&'life0 self,
prompt: &'life1 str,
max_tokens: usize,
) -> Pin<Box<dyn Future<Output = Result<String, Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
'life1: 'async_trait,
TorchEngine: 'async_trait,
Source§fn generate_with_params<'life0, 'async_trait>(
&'life0 self,
request: GenerationRequest,
) -> Pin<Box<dyn Future<Output = Result<GenerationResult, Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
TorchEngine: 'async_trait,
fn generate_with_params<'life0, 'async_trait>(
&'life0 self,
request: GenerationRequest,
) -> Pin<Box<dyn Future<Output = Result<GenerationResult, Error>> + Send + 'async_trait>>where
'life0: 'async_trait,
TorchEngine: 'async_trait,
Source§fn model_info(&self) -> ModelInfo
fn model_info(&self) -> ModelInfo
Auto Trait Implementations§
impl Freeze for TorchEngine
impl !RefUnwindSafe for TorchEngine
impl Send for TorchEngine
impl Sync for TorchEngine
impl Unpin for TorchEngine
impl UnsafeUnpin for TorchEngine
impl !UnwindSafe for TorchEngine
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request