Skip to main content

InferenceServiceConfig

Struct InferenceServiceConfig 

Source
pub struct InferenceServiceConfig { /* private fields */ }
Expand description

Configuration for spawning an InferenceService.

This struct holds all parameters needed to initialize an InferenceService. It is Send and can be moved to a dedicated thread via Spawnable::run(). The actual GPU initialization happens on the service thread.

Implementations§

Source§

impl InferenceServiceConfig

Source

pub fn new( model_path: impl AsRef<Path>, config: RuntimeConfig, server_pubkey: VerifyingKey, signing_key: SigningKey, transport: TransportConfig, policy_transport: TransportConfig, fs: Option<WorktreeClient>, ) -> Self

Create a new InferenceServiceConfig.

No GPU work is done here — all heavy initialization is deferred to the service thread via Spawnable::run().

Important: Does NOT take a PolicyClient. The policy client is created lazily inside run() on the service thread’s own Tokio runtime, because ZMQ sockets must be registered with the correct runtime’s reactor.

Source

pub fn with_pinned_artifact(self, artifact: Arc<PinnedModelArtifact>) -> Self

Bind all base-model path reads to the owned exact-tree projection. Adapter and training writes continue to use the original checkout.

Source

pub fn with_instance_identity( self, service_name: String, tenant_domain: String, controller_pubkey: VerifyingKey, ) -> Self

Bind this engine to one verified tenant and one controller identity.

Source

pub fn with_model_ref(self, model_ref: impl Into<String>) -> Self

Bind the model reference this pinned instance serves. Present ⇒ controller internal work orders (iw+jwt) are admissible; absent ⇒ they are never admitted (standalone direct-caller service).

Source

pub fn with_incarnation_handoff( self, sender: Sender<IncarnationHandoff>, attempt_generation: u64, ) -> Self

Attach this spawn attempt’s private incarnation handoff. The worker sends its worker-generated incarnation + audience + attempt identity after its engine is initialized; the owner awaits it after readiness.

Source

pub fn with_stream_plane( self, producer_reach_config: ProducerReachConfigHandle, moq_origin: MoqStreamOriginHandle, ) -> Self

Publish generated streams through an outer service’s QUIC/MoQ relay.

Source

pub fn with_quic_config(self, quic_config: Option<QuicLoopConfig>) -> Self

Bind a browser-addressable RPC/MoQ endpoint owned by this process.

Source

pub fn with_quic_advertise_addr( self, advertise_addr: Option<SocketAddr>, ) -> Self

Set the externally routable address signed into StreamInfo.

Source

pub fn with_expected_audience(self, audience: String) -> Self

Set the expected audience for JWT validation.

Source

pub fn with_jwt_key_source(self, src: Arc<dyn JwtKeySource>) -> Self

Set the JWT key source for verifying JWTs (local and federated).

Source

pub fn network_reach_handle(&self) -> Arc<RwLock<Option<TransportConfig>>> ⓘ

Trait Implementations§

Source§

impl Spawnable for InferenceServiceConfig

Source§

fn name(&self) -> &str

Service name (for logging and registry).
Source§

fn registrations(&self) -> Vec<(SocketKind, TransportConfig)>

Endpoints to register with EndpointRegistry. Read more
Source§

fn run( self: Box<Self>, shutdown: Arc<Notify>, on_ready: Option<Sender<()>>, ) -> Result<()>

Run the service on current thread (blocking). Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
§

impl<T> Any for T
where T: Any,

§

fn into_any(self: Box<T>) -> Box<dyn Any>

§

fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>

§

fn type_name(&self) -> &'static str

§

impl<'a, T, E> AsTaggedExplicit<'a, E> for T
where T: 'a,

§

fn explicit(self, class: Class, tag: u32) -> TaggedParser<'a, Explicit, Self, E>

§

impl<'a, T, E> AsTaggedImplicit<'a, E> for T
where T: 'a,

§

fn implicit( self, class: Class, constructed: bool, tag: u32, ) -> TaggedParser<'a, Implicit, Self, E>

Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> FutureExt for T

§

fn with_context(self, otel_cx: Context) -> WithContext<Self>

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
§

fn with_current_context(self) -> WithContext<Self>

Attaches the current Context to this type, returning a WithContext wrapper. Read more
§

impl<T, W> HasTypeWitness<W> for T
where W: MakeTypeWitness<Arg = T>, T: ?Sized,

§

const WITNESS: W = W::MAKE

A constant of the type witness
§

impl<T> Identity for T
where T: ?Sized,

§

const TYPE_EQ: TypeEq<T, <T as Identity>::Type> = TypeEq::NEW

Proof that Self is the same type as Self::Type, provides methods for casting between Self and Self::Type.
§

type Type = T

The same type as Self, used to emulate type equality bounds (T == U) with associated type equality constraints (T: Identity<Type = U>).
§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self>

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self>

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self>

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
§

impl<T> JaggedSlice<T> for T

§

type Index = Index2

§

fn push_into(self, array: &mut Jagged<T>)

§

fn insert_into(self, index: <T as JaggedSlice<T>>::Index, array: &mut Jagged<T>)

§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
§

impl<T> ServiceExt for T

§

fn map_response_body<F>(self, f: F) -> MapResponseBody<Self, F>
where Self: Sized,

Apply a transformation to the response body. Read more
§

fn trace_for_http(self) -> Trace<Self, SharedClassifier<ServerErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using HTTP status codes. Read more
§

fn trace_for_grpc(self) -> Trace<Self, SharedClassifier<GrpcErrorsAsFailures>>
where Self: Sized,

High level tracing that classifies responses using gRPC headers. Read more
§

fn follow_redirects(self) -> FollowRedirect<Self>
where Self: Sized,

Follow redirect resposes using the Standard policy. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self>

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

impl<T> ErasedDestructor for T
where T: 'static,

§

impl<T> MaybeSend for T
where T: Send,

§

impl<T> MaybeSend for T
where T: Send,