Skip to main content

Module inference

Module inference 

Source
Expand description

ZMQ-based inference service for text generation

This service wraps TorchEngine and provides a ZMQ interface for inference operations. It uses:

  • REQ/REP for standard requests (generate, model_info, lora operations, etc.)
  • PUB (via StreamService) for streaming generation with JWT authorization

§Thread Model

The service runs on a dedicated thread with its own TorchEngine because:

  • tch-rs types contain raw pointers (not Send)
  • GPU operations benefit from thread isolation

§Streaming Architecture

Client                    InferenceService         StreamService
  │                            │                         │
  │─ REQ: GenerateStream ─────►│                         │
  │◄─ REP: {stream_id,endpoint}│                         │
  │                            │                         │
  │                            │── PUB: chunks ─────────►│ (validates JWT)
  │                            │                         │
  │  SUB: stream-{id}|{jwt} ───────────────────────────►│
  │◄────────────────── chunks (JWT stripped) ───────────│

InferenceService publishes to StreamService’s XSUB (PUB socket). StreamService validates JWT at subscription and forwards to clients.

§Authorization

Uses InferenceHandler::authorize() via generated dispatch for policy-backed authorization on all requests. The handler delegates to PolicyClient.

Structs§

IncarnationHandoff
Private per-attempt readiness handoff from the worker to its owning ModelService (Sol bounded incarnation plan). The worker generates its own 256-bit incarnation inside its service thread and reports the exact versioned audience plus the spawn-attempt identity through this channel — never via a request, resolver record, or cached deterministic name.
InferenceService
ZMQ-based inference service
InferenceServiceConfig
Configuration for spawning an InferenceService.
InferenceServiceInner
ZMQ-based inference service

Enums§

StreamChunkMessage
Message received from StreamService during streaming generation.

Constants§

INCARNATION_HANDOFF_TIMEOUT_SECS
Seconds a ModelService waits for the worker’s incarnation handoff after the spawner reports readiness. Missing readiness fails the load closed.
INFERENCE_ENDPOINT
Default endpoint for the inference service