pub enum InferenceRequest {
Show 16 variants
Generate {
request: GenerationRequest,
reply: Sender<Result<GenerationResult, InferenceError>>,
},
GenerateStream {
request: GenerationRequest,
chunk_sender: Sender<Result<String, InferenceError>>,
stats_sender: Sender<StreamStats>,
reply: Sender<Result<(), InferenceError>>,
},
ModelInfo {
reply: Sender<Result<ModelInfo, InferenceError>>,
},
IsReady {
reply: Sender<Result<bool, InferenceError>>,
},
ApplyChatTemplate {
messages: Vec<ChatMessage>,
add_generation_prompt: bool,
tools: Option<Value>,
reply: Sender<Result<String, InferenceError>>,
},
CreateLora {
config: TenantDeltaConfig,
reply: Sender<Result<(), InferenceError>>,
},
LoadLora {
path: PathBuf,
reply: Sender<Result<(), InferenceError>>,
},
SaveLora {
path: PathBuf,
reply: Sender<Result<(), InferenceError>>,
},
UnloadLora {
reply: Sender<Result<(), InferenceError>>,
},
HasLora {
reply: Sender<Result<bool, InferenceError>>,
},
SetSession {
session_id: String,
reply: Sender<Result<(), InferenceError>>,
},
ClearSession {
reply: Sender<Result<(), InferenceError>>,
},
ReleaseSession {
session_id: String,
reply: Sender<Result<(), InferenceError>>,
},
Embed {
images: Vec<Vec<u8>>,
reply: Sender<Result<Vec<Vec<f32>>, InferenceError>>,
},
HealthCheck {
reply: Sender<Result<(), InferenceError>>,
},
Shutdown {
reply: Sender<Result<(), InferenceError>>,
},
}Expand description
Request types for the inference service.
All variants include a oneshot reply channel for returning results. For streaming generation, we use mpsc channels for chunks.
Variants§
Generate
Non-streaming generation request.
GenerateStream
Streaming generation request.
Unlike other requests, this uses an mpsc channel for streaming chunks and a separate oneshot for final stats.
Fields
request: GenerationRequestchunk_sender: Sender<Result<String, InferenceError>>Channel to send text chunks as they’re generated.
stats_sender: Sender<StreamStats>Channel to send final statistics after generation completes.
reply: Sender<Result<(), InferenceError>>Reply channel for acknowledging request receipt or early errors.
ModelInfo
Get model information.
Fields
reply: Sender<Result<ModelInfo, InferenceError>>IsReady
Check if model is ready.
Fields
reply: Sender<Result<bool, InferenceError>>ApplyChatTemplate
Apply chat template to messages.
CreateLora
Create LoRA adapter structure.
LoadLora
Load LoRA adapter from file.
SaveLora
Save LoRA adapter to file.
UnloadLora
Unload current LoRA adapter.
Fields
reply: Sender<Result<(), InferenceError>>HasLora
Check if LoRA is loaded.
Fields
reply: Sender<Result<bool, InferenceError>>SetSession
Set active session for KV cache isolation.
ClearSession
Clear current session’s KV cache.
Fields
reply: Sender<Result<(), InferenceError>>ReleaseSession
Release a session’s KV cache.
Embed
Compute embeddings for images.
HealthCheck
Health check request.
Fields
reply: Sender<Result<(), InferenceError>>Shutdown
Shutdown request.
Fields
reply: Sender<Result<(), InferenceError>>Trait Implementations§
Auto Trait Implementations§
impl Freeze for InferenceRequest
impl !RefUnwindSafe for InferenceRequest
impl Send for InferenceRequest
impl Sync for InferenceRequest
impl Unpin for InferenceRequest
impl UnsafeUnpin for InferenceRequest
impl !UnwindSafe for InferenceRequest
Blanket Implementations§
§impl<T> AnySync for T
impl<T> AnySync for T
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request