pub struct InferClient { /* private fields */ }Expand description
Scoped client for infer operations (transport-agnostic).
Implementations§
Source§impl InferClient
impl InferClient
Sourcepub async fn call(&self, payload: Vec<u8>) -> Result<Vec<u8>>
pub async fn call(&self, payload: Vec<u8>) -> Result<Vec<u8>>
Send a raw request and return the raw response bytes.
Sourcepub async fn call_streaming(
&self,
payload: Vec<u8>,
ephemeral_pubkey: [u8; 32],
) -> Result<Vec<u8>>
pub async fn call_streaming( &self, payload: Vec<u8>, ephemeral_pubkey: [u8; 32], ) -> Result<Vec<u8>>
Send a streaming request with ephemeral DH pubkey.
Sourcepub fn parse_scoped_response(bytes: &[u8]) -> Result<InferClientResponseVariant>
pub fn parse_scoped_response(bytes: &[u8]) -> Result<InferClientResponseVariant>
Parse a scoped response from raw bytes.
Sourcepub async fn generate_stream(
&self,
__request: &GenerationRequest,
ephemeral_pubkey: [u8; 32],
) -> Result<StreamInfo>
pub async fn generate_stream( &self, __request: &GenerationRequest, ephemeral_pubkey: [u8; 32], ) -> Result<StreamInfo>
Run inference with automatic domain adaptation. When TTT is enabled, the model adapts to your prompt BEFORE generating — the response is always produced using the adapted weights, even when adaptationStrategy=speculative. Check onlineTrainingMetrics.recommendation in the response, then call ttt.writeback (if true) or ttt.evict (if false) within the pending rollback window (default 60 seconds). If you call generateStream again before resolving a pending adaptation, the new adaptation stacks on top and evict will restore to the state before the first pending adaptation. Pending adaptations auto-evict after the timeout if writeback/evict is not called.
Scope: infer
Sourcepub async fn apply_chat_template(
&self,
__request: &ChatTemplateRequest,
) -> Result<String>
pub async fn apply_chat_template( &self, __request: &ChatTemplateRequest, ) -> Result<String>
Apply chat template to messages for a loaded model
Scope: query
Sourcepub async fn status(&self) -> Result<ModelStatusResponse>
pub async fn status(&self) -> Result<ModelStatusResponse>
Get detailed status information about a model including online training configuration
Scope: query
Sourcepub async fn embed(
&self,
__request: &EmbedImagesRequest,
) -> Result<EmbedImagesResponse>
pub async fn embed( &self, __request: &EmbedImagesRequest, ) -> Result<EmbedImagesResponse>
Compute embeddings for one or more images. Returns embedding vectors from the model’s vision encoder (e.g. SigLIP). Synchronous — returns all embeddings in a single response.
Scope: infer
Source§impl InferClient
impl InferClient
Source§impl InferClient
impl InferClient
Trait Implementations§
Source§impl Clone for InferClient
impl Clone for InferClient
Source§fn clone(&self) -> InferClient
fn clone(&self) -> InferClient
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read moreSource§impl InferRpc for InferClient
impl InferRpc for InferClient
fn generate_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
request: &'life1 GenerationRequest,
) -> Pin<Box<dyn Future<Output = Result<MoqStreamHandle>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn apply_chat_template<'life0, 'life1, 'async_trait>(
&'life0 self,
request: &'life1 ChatTemplateRequest,
) -> Pin<Box<dyn Future<Output = Result<String>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
fn status<'life0, 'async_trait>(
&'life0 self,
) -> Pin<Box<dyn Future<Output = Result<ModelStatusResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
fn embed<'life0, 'life1, 'async_trait>(
&'life0 self,
request: &'life1 EmbedImagesRequest,
) -> Pin<Box<dyn Future<Output = Result<EmbedImagesResponse>> + Send + 'async_trait>>where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Auto Trait Implementations§
impl Freeze for InferClient
impl !RefUnwindSafe for InferClient
impl Send for InferClient
impl Sync for InferClient
impl Unpin for InferClient
impl UnsafeUnpin for InferClient
impl !UnwindSafe for InferClient
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more