Skip to main content

Module model

Module model 

Source
Expand description

Model service for managing InferenceService instances over ZMQ

This service manages the lifecycle of InferenceService instances. It handles model loading, unloading, and routes inference requests to the appropriate InferenceService based on model reference.

§Architecture

REST API / CLI
      │
      │ ModelClient (async ZMQ I/O)
      ▼
ModelService (multi-threaded runtime)
      │
      ├── LRU cache of loaded models
      ├── Spawns InferenceService per model
      └── Routes requests to InferenceService
            │
            │ InferenceClient (async ZMQ I/O)
            ▼
      InferenceService (dedicated thread per model)

§Endpoint

Uses registry().endpoint("model", SocketKind::Rep) for the REP endpoint. Default fallback: inproc://hyprstream/model

Structs§

LoadedModel
Information about a loaded model
ModelLoadKey
Per-load-attempt key into the model-service TerminalStore (EV7/#649).
ModelService
Model service that manages InferenceService lifecycle.
ModelServiceConfig
Model service configuration
ModelServiceInner
Inner state for ModelService, behind Arc for continuation capture.
StagingModelPin

Enums§

ModelLoadTerminal
Terminal payload for a model load attempt (EV7/#649). Latched host-side by ModelService when a load completes — Loaded on success, LoadFailed otherwise — so a late load --wait (or a future P9 /model/<ref>/loaded file) is served the retained value instead of missing the live model.lifecycle edge it subscribed to too late.
ModelRefDispatch
Prefix-dispatch arm for the modelRef grammar (#395).

Constants§

MODEL_ENDPOINT
Default endpoint for the model service

Functions§

model_ref_dispatch
Classify a modelRef string into its grammar arm (#395 prefix dispatch).