Expand description
Leaf-tier cell-router for federated inference scheduling (#322).
Capacity-weighted HRW (Highest Random Weight / Rendezvous Hashing) for session placement within a single cell (a PDS / trust domain). This is the leaf tier of the federated scheduling design — a bounded set of tens to low-thousands of nodes. The global tier (cell directory over ~10k–100k cells) is a separate epic.
§Two keys, two jobs
- OID (git commit SHA) = content selector → yields the replica set
(nodes serving that checkpoint). The router body operates on a resolved
replica set; how the OID is resolved (
at://→DID→OID per the federated-model-addressing spike, or the localResolver) is the entry’s job, not the router’s. - session_id = placement key → capacity-weighted HRW over the replica set → owner node.
HRW gives both placement properties simultaneously:
- Affinity: the same
session_idalways hashes to the same node while the replica set is stable → KV-cache reuse. - Spread: distinct
session_ids distribute uniformly across the set.
§Failure detection
Two reactive layers, no proactive membership protocol (SWIM is unnecessary at this tier — see the federated-scheduling-dht-pq-spike §4):
- Dial-fail (sync): a failed dial marks the node down and reassigns to the next HRW choice on the next call.
- Token-stream stall watchdog: absence of tokens past a deadline marks the node stalled and reassigns.
§v1 constraints
- No new capnp surface. Reuse
healthCheck@13/isReady@3fromservice_events.capnpfor health integration. - No gossip. Load counters are router-side local state only (v1); gossip is a future option for proactive capacity rebalancing (spike §4).
- CPU-testable. Router logic is plain data structures — no tensors, no
GPU. tch-rs is
!Send; this module touches none of it.
§Determinism
HRW requires a hash that is stable across processes (so two routers in
the same cell pick the same owner for the same session). DefaultHasher::new()
is documented to use constant keys, so its output is deterministic across
runs and processes. We do not use RandomState (which is randomly seeded
per process and would break cross-router agreement).
Structs§
- Cell
Router - Router-side load + health state for a cell’s replica set.
- Inference
Server Info - Information about a candidate inference server in the cell.
- Placement
- Outcome of a placement decision.
- Replica
Id - Opaque identifier for an inference replica used only for placement.
- Session
Affinity - Session→owner affinity map with heartbeat-lease renewal.
Enums§
- Exclude
Reason - Reason a node was excluded from placement (for the failure log / tests).
- Placement
Policy - Where to place a session within the replica set.
Constants§
- DEFAULT_
DOWN_ TTL - Default negative-cache TTL on a dial-fail (avoid hammering a down node).
- DEFAULT_
LEASE_ DURATION - Default lease for a session→owner affinity binding. Renewed on every heartbeat; expiry forces a fresh HRW on the next request.
- DEFAULT_
STALL_ DEADLINE - Default deadline before a token-starved stream is declared stalled.
Functions§
- hrw_
select - Highest Random Weight (Rendezvous Hashing) selection.
- hrw_
select_ skip - Select the
skip-th HRW choice (0 = primary, 1 = first fallback, …). - least_
loaded_ select - Select the least-loaded candidate (lowest
active_sessions, ties broken by most free GPU memory). ReturnsNonefor an empty set.
Type Aliases§
- Session
Id - Identifier for a chat / inference session (used as the HRW placement key).