pub struct InferenceStage { /* private fields */ }Expand description
A loaded, same-thread decoder stage.
Construction is the production consumer of ModelFactory::create_stage.
One instance is deliberately bound to one StageSequence and therefore
one model-local cache timeline. The object intentionally has no transport
implementation: callers encode activations at a process or host boundary.
Implementations§
Source§impl InferenceStage
impl InferenceStage
Sourcepub async fn load(
model_path: &Path,
device: &Device,
dtype: Kind,
max_context: Option<usize>,
kv_quant_type: KVQuantType,
request: ModelStageRequest,
fp8_dequant_load: bool,
) -> Result<InferenceStage, Error>
pub async fn load( model_path: &Path, device: &Device, dtype: Kind, max_context: Option<usize>, kv_quant_type: KVQuantType, request: ModelStageRequest, fp8_dequant_load: bool, ) -> Result<InferenceStage, Error>
Load exactly the weights required for one contiguous stage.
fp8_dequant_load forwards the caller’s explicit FP8 loading policy to
the model factory; stage loading does not change that policy.
Sourcepub fn contract(&self) -> &StageContract
pub fn contract(&self) -> &StageContract
Immutable stage role and range contract.
Sourcepub fn execute(
&mut self,
sequence: &mut StageSequence,
input: StageInput<'_>,
start_pos: usize,
) -> Result<StageOutput, Error>
pub fn execute( &mut self, sequence: &mut StageSequence, input: StageInput<'_>, start_pos: usize, ) -> Result<StageOutput, Error>
Execute one prefill or decode step for this stage.
sequence is the sole cache timeline admitted by this stage instance.
start_pos must advance monotonically from zero for that sequence. The
loaded model keeps its per-layer KV (and architecture-specific recurrent)
state locally; the typed result stays on the engine-owning thread.
A failure after decoder execution starts can leave that cache partially
updated. Such a stage rejects further execution and must be reloaded
with a fresh sequence. A sequence cannot migrate to a different stage,
whose model-local cache would not contain that sequence’s history.
Auto Trait Implementations§
impl Freeze for InferenceStage
impl !RefUnwindSafe for InferenceStage
impl !Send for InferenceStage
impl !Sync for InferenceStage
impl Unpin for InferenceStage
impl UnsafeUnpin for InferenceStage
impl !UnwindSafe for InferenceStage
Blanket Implementations§
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request