pub struct InferenceService { /* private fields */ }Expand description
ZMQ-based inference service
Wraps InferenceServiceInner in Arc for continuation-based streaming.
All field and method access is transparent via Deref.
Implementations§
Source§impl InferenceService
impl InferenceService
Sourcepub async fn drain_export(&self) -> Result<usize>
pub async fn drain_export(&self) -> Result<usize>
Drain-time export hook (#869): force-snapshot every resident per-tenant delta
with accumulated steps to adapters/.snapshots/, so a graceful shutdown / pod
preStop / scale-in does not drop uncommitted TTT adaptation.
This is the reachable seam for the #865 serving-controller drain path to call.
It is exposed rather than auto-wired into Spawnable::run’s shutdown Notify
because the delta pool lives inside this !Send service behind the
LocalServiceBridge — there is no post-shutdown hook on the bridge lifecycle
today to invoke it from (adding one is part of the #865 rework, not this change).
Returns the number of deltas persisted (a None snapshot slot is a loss event and
is logged by the pool).
Sourcepub async fn load_lora(&self, path: &Path) -> Result<()>
pub async fn load_lora(&self, path: &Path) -> Result<()>
Load a PEFT adapter directory as the base delta.
path is a PEFT adapter directory (e.g. adapters/my-adapter) containing
adapter_model.safetensors. The loaded adapter is stored as base_delta
and applied to all inference requests. If a per-tenant TTT delta also exists,
the two are composed (corrections summed) during inference via resolve_delta().
Sourcepub async fn save_lora(&self, path: &str) -> Result<()>
pub async fn save_lora(&self, path: &str) -> Result<()>
Save the current base delta to a safetensors file.
Sourcepub async fn unload_lora(&self) -> Result<()>
pub async fn unload_lora(&self) -> Result<()>
Unload the current base LoRA adapter.
Sourcepub async fn merge_lora(
&self,
path: &Path,
weight: f32,
strategy_name: &str,
) -> Result<()>
pub async fn merge_lora( &self, path: &Path, weight: f32, strategy_name: &str, ) -> Result<()>
Merge an on-disk PEFT adapter into the currently loaded base_delta.
Reads the adapter from disk via WorktreeClient, loads it as a TenantDelta, then merges its corrections into the existing base_delta using the specified merge strategy. Requires a base_delta to already be loaded.
Trait Implementations§
Source§impl Clone for InferenceService
impl Clone for InferenceService
Source§impl Deref for InferenceService
impl Deref for InferenceService
Source§impl InferenceHandler for InferenceService
impl InferenceHandler for InferenceService
Source§fn handle_generate_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 GenerationRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_generate_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 GenerationRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
generateStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_model_info<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_model_info<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
modelInfo request.Source§fn handle_is_ready<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_is_ready<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
isReady request.Source§fn handle_get_layer_profile<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_get_layer_profile<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
getLayerProfile request.Source§fn handle_apply_chat_template<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ChatTemplateRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_apply_chat_template<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ChatTemplateRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
applyChatTemplate request.Source§fn handle_create_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 LoraConfig,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_create_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 LoraConfig,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
createLora request.Source§fn handle_load_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_load_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
loadLora request.Source§fn handle_save_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_save_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
saveLora request.Source§fn handle_unload_lora<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_unload_lora<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
unloadLora request.Source§fn handle_has_lora<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_has_lora<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
hasLora request.Source§fn handle_set_session<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_set_session<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
setSession request.Source§fn handle_clear_session<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_clear_session<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
clearSession request.Source§fn handle_release_session<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_release_session<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
releaseSession request.Source§fn handle_health_check<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_health_check<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
healthCheck request.Source§fn handle_shutdown<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_shutdown<'life0, 'life1, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
shutdown request.Source§fn handle_ttt_writeback<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_ttt_writeback<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
tttWriteback request.Source§fn handle_ttt_evict<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_ttt_evict<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
tttEvict request.Source§fn handle_train_step<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
_data: &'life2 TrainStepRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_train_step<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
_data: &'life2 TrainStepRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
trainStep request.Source§fn handle_train_step_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 TrainStepRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_train_step_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 TrainStepRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
trainStepStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_ttt_zero<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_ttt_zero<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
tttZero request.Source§fn handle_get_delta_status<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_get_delta_status<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
getDeltaStatus request.Source§fn handle_save_adaptation<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 SaveAdaptationRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_save_adaptation<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 SaveAdaptationRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
saveAdaptation request.Source§fn handle_snapshot_delta<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_snapshot_delta<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
snapshotDelta request.Source§fn handle_export_peft_adapter<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ExportPeftRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_export_peft_adapter<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ExportPeftRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
exportPeftAdapter request.Source§fn handle_merge_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 MergeLoraRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_merge_lora<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 MergeLoraRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
mergeLora request.Source§fn handle_create_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 LoraConfig,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_create_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 LoraConfig,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
createLoraStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_load_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_load_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
loadLoraStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_save_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_save_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
value: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
saveLoraStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_save_adaptation_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 SaveAdaptationRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_save_adaptation_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 SaveAdaptationRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
saveAdaptationStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_snapshot_delta_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn handle_snapshot_delta_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
snapshotDeltaStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_export_peft_adapter_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ExportPeftRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_export_peft_adapter_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 ExportPeftRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
exportPeftAdapterStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_merge_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 MergeLoraRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_merge_lora_stream<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 MergeLoraRequest,
) -> Pin<Box<dyn Future<Output = Result<(StreamInfo, Continuation)>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
mergeLoraStream streaming request. Returns (StreamInfo, mandatory continuation).Source§fn handle_embed<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 EmbedImagesRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
fn handle_embed<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
_ctx: &'life1 EnvelopeContext,
_request_id: u64,
data: &'life2 EmbedImagesRequest,
) -> Pin<Box<dyn Future<Output = Result<InferenceResponseVariant>> + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
embed request.Auto Trait Implementations§
impl Freeze for InferenceService
impl !RefUnwindSafe for InferenceService
impl Send for InferenceService
impl Sync for InferenceService
impl Unpin for InferenceService
impl UnsafeUnpin for InferenceService
impl !UnwindSafe for InferenceService
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request