pub struct InferenceServiceConfig { /* private fields */ }Expand description
Configuration for spawning an InferenceService.
This struct holds all parameters needed to initialize an InferenceService.
It is Send and can be moved to a dedicated thread via Spawnable::run().
The actual GPU initialization happens on the service thread.
Implementations§
Source§impl InferenceServiceConfig
impl InferenceServiceConfig
Sourcepub fn new(
model_path: impl AsRef<Path>,
config: RuntimeConfig,
server_pubkey: VerifyingKey,
signing_key: SigningKey,
transport: TransportConfig,
policy_transport: TransportConfig,
fs: Option<WorktreeClient>,
) -> Self
pub fn new( model_path: impl AsRef<Path>, config: RuntimeConfig, server_pubkey: VerifyingKey, signing_key: SigningKey, transport: TransportConfig, policy_transport: TransportConfig, fs: Option<WorktreeClient>, ) -> Self
Create a new InferenceServiceConfig.
No GPU work is done here — all heavy initialization is deferred
to the service thread via Spawnable::run().
Important: Does NOT take a PolicyClient. The policy client is created
lazily inside run() on the service thread’s own Tokio runtime, because
ZMQ sockets must be registered with the correct runtime’s reactor.
Sourcepub fn with_pinned_artifact(self, artifact: Arc<PinnedModelArtifact>) -> Self
pub fn with_pinned_artifact(self, artifact: Arc<PinnedModelArtifact>) -> Self
Bind all base-model path reads to the owned exact-tree projection. Adapter and training writes continue to use the original checkout.
Sourcepub fn with_instance_identity(
self,
service_name: String,
tenant_domain: String,
controller_pubkey: VerifyingKey,
) -> Self
pub fn with_instance_identity( self, service_name: String, tenant_domain: String, controller_pubkey: VerifyingKey, ) -> Self
Bind this engine to one verified tenant and one controller identity.
Sourcepub fn with_model_ref(self, model_ref: impl Into<String>) -> Self
pub fn with_model_ref(self, model_ref: impl Into<String>) -> Self
Bind the model reference this pinned instance serves. Present ⇒
controller internal work orders (iw+jwt) are admissible; absent ⇒
they are never admitted (standalone direct-caller service).
Sourcepub fn with_incarnation_handoff(
self,
sender: Sender<IncarnationHandoff>,
attempt_generation: u64,
) -> Self
pub fn with_incarnation_handoff( self, sender: Sender<IncarnationHandoff>, attempt_generation: u64, ) -> Self
Attach this spawn attempt’s private incarnation handoff. The worker sends its worker-generated incarnation + audience + attempt identity after its engine is initialized; the owner awaits it after readiness.
Sourcepub fn with_stream_plane(
self,
producer_reach_config: ProducerReachConfigHandle,
moq_origin: MoqStreamOriginHandle,
) -> Self
pub fn with_stream_plane( self, producer_reach_config: ProducerReachConfigHandle, moq_origin: MoqStreamOriginHandle, ) -> Self
Publish generated streams through an outer service’s QUIC/MoQ relay.
Sourcepub fn with_quic_config(self, quic_config: Option<QuicLoopConfig>) -> Self
pub fn with_quic_config(self, quic_config: Option<QuicLoopConfig>) -> Self
Bind a browser-addressable RPC/MoQ endpoint owned by this process.
Sourcepub fn with_quic_advertise_addr(
self,
advertise_addr: Option<SocketAddr>,
) -> Self
pub fn with_quic_advertise_addr( self, advertise_addr: Option<SocketAddr>, ) -> Self
Set the externally routable address signed into StreamInfo.
Sourcepub fn with_expected_audience(self, audience: String) -> Self
pub fn with_expected_audience(self, audience: String) -> Self
Set the expected audience for JWT validation.
Sourcepub fn with_jwt_key_source(self, src: Arc<dyn JwtKeySource>) -> Self
pub fn with_jwt_key_source(self, src: Arc<dyn JwtKeySource>) -> Self
Set the JWT key source for verifying JWTs (local and federated).
pub fn network_reach_handle(&self) -> Arc<RwLock<Option<TransportConfig>>> ⓘ
Trait Implementations§
Source§impl Spawnable for InferenceServiceConfig
impl Spawnable for InferenceServiceConfig
Source§fn registrations(&self) -> Vec<(SocketKind, TransportConfig)>
fn registrations(&self) -> Vec<(SocketKind, TransportConfig)>
Auto Trait Implementations§
impl Freeze for InferenceServiceConfig
impl !RefUnwindSafe for InferenceServiceConfig
impl Send for InferenceServiceConfig
impl !Sync for InferenceServiceConfig
impl Unpin for InferenceServiceConfig
impl UnsafeUnpin for InferenceServiceConfig
impl !UnwindSafe for InferenceServiceConfig
Blanket Implementations§
§impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedExplicit<'a, E> for Twhere
T: 'a,
§impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
impl<'a, T, E> AsTaggedImplicit<'a, E> for Twhere
T: 'a,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
§impl<T> FutureExt for T
impl<T> FutureExt for T
§fn with_context(self, otel_cx: Context) -> WithContext<Self>
fn with_context(self, otel_cx: Context) -> WithContext<Self>
§fn with_current_context(self) -> WithContext<Self>
fn with_current_context(self) -> WithContext<Self>
§impl<T> Identity for Twhere
T: ?Sized,
impl<T> Identity for Twhere
T: ?Sized,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request