Expand description
Model service for managing InferenceService instances over ZMQ
This service manages the lifecycle of InferenceService instances. It handles model loading, unloading, and routes inference requests to the appropriate InferenceService based on model reference.
§Architecture
REST API / CLI
│
│ ModelClient (async ZMQ I/O)
▼
ModelService (multi-threaded runtime)
│
├── LRU cache of loaded models
├── Spawns InferenceService per model
└── Routes requests to InferenceService
│
│ InferenceClient (async ZMQ I/O)
▼
InferenceService (dedicated thread per model)§Endpoint
Uses registry().endpoint("model", SocketKind::Rep) for the REP endpoint.
Default fallback: inproc://hyprstream/model
Structs§
- Loaded
Model - Information about a loaded model
- Model
Load Key - Per-load-attempt key into the model-service
TerminalStore(EV7/#649). - Model
Service - Model service that manages InferenceService lifecycle.
- Model
Service Config - Model service configuration
- Model
Service Inner - Inner state for ModelService, behind Arc for continuation capture.
- Staging
Model Pin
Enums§
- Model
Load Terminal - Terminal payload for a model load attempt (EV7/#649). Latched host-side by
ModelServicewhen a load completes —Loadedon success,LoadFailedotherwise — so a lateload --wait(or a future P9/model/<ref>/loadedfile) is served the retained value instead of missing the livemodel.lifecycleedge it subscribed to too late. - Model
RefDispatch - Prefix-dispatch arm for the
modelRefgrammar (#395).
Constants§
- MODEL_
ENDPOINT - Default endpoint for the model service
Functions§
- model_
ref_ dispatch - Classify a
modelRefstring into its grammar arm (#395 prefix dispatch).