Expand description
ZMQ-based inference service for text generation
This service wraps TorchEngine and provides a ZMQ interface for inference operations. It uses:
- REQ/REP for standard requests (generate, model_info, lora operations, etc.)
- PUB (via StreamService) for streaming generation with JWT authorization
§Thread Model
The service runs on a dedicated thread with its own TorchEngine because:
- tch-rs types contain raw pointers (not Send)
- GPU operations benefit from thread isolation
§Streaming Architecture
Client InferenceService StreamService
│ │ │
│─ REQ: GenerateStream ─────►│ │
│◄─ REP: {stream_id,endpoint}│ │
│ │ │
│ │── PUB: chunks ─────────►│ (validates JWT)
│ │ │
│ SUB: stream-{id}|{jwt} ───────────────────────────►│
│◄────────────────── chunks (JWT stripped) ───────────│InferenceService publishes to StreamService’s XSUB (PUB socket). StreamService validates JWT at subscription and forwards to clients.
§Authorization
Uses InferenceHandler::authorize() via generated dispatch for policy-backed
authorization on all requests. The handler delegates to PolicyClient.
Structs§
- Incarnation
Handoff - Private per-attempt readiness handoff from the worker to its owning ModelService (Sol bounded incarnation plan). The worker generates its own 256-bit incarnation inside its service thread and reports the exact versioned audience plus the spawn-attempt identity through this channel — never via a request, resolver record, or cached deterministic name.
- Inference
Service - ZMQ-based inference service
- Inference
Service Config - Configuration for spawning an InferenceService.
- Inference
Service Inner - ZMQ-based inference service
Enums§
- Stream
Chunk Message - Message received from StreamService during streaming generation.
Constants§
- INCARNATION_
HANDOFF_ TIMEOUT_ SECS - Seconds a ModelService waits for the worker’s incarnation handoff after the spawner reports readiness. Missing readiness fails the load closed.
- INFERENCE_
ENDPOINT - Default endpoint for the inference service