Skip to main content

Module router

Module router 

Source
Expand description

Leaf-tier cell-router for federated inference scheduling (#322).

Capacity-weighted HRW (Highest Random Weight / Rendezvous Hashing) for session placement within a single cell (a PDS / trust domain). This is the leaf tier of the federated scheduling design — a bounded set of tens to low-thousands of nodes. The global tier (cell directory over ~10k–100k cells) is a separate epic.

§Two keys, two jobs

  • OID (git commit SHA) = content selector → yields the replica set (nodes serving that checkpoint). The router body operates on a resolved replica set; how the OID is resolved (at://→DID→OID per the federated-model-addressing spike, or the local Resolver) is the entry’s job, not the router’s.
  • session_id = placement key → capacity-weighted HRW over the replica set → owner node.

HRW gives both placement properties simultaneously:

  • Affinity: the same session_id always hashes to the same node while the replica set is stable → KV-cache reuse.
  • Spread: distinct session_ids distribute uniformly across the set.

§Failure detection

Two reactive layers, no proactive membership protocol (SWIM is unnecessary at this tier — see the federated-scheduling-dht-pq-spike §4):

  • Dial-fail (sync): a failed dial marks the node down and reassigns to the next HRW choice on the next call.
  • Token-stream stall watchdog: absence of tokens past a deadline marks the node stalled and reassigns.

§v1 constraints

  • No new capnp surface. Reuse healthCheck@13 / isReady@3 from service_events.capnp for health integration.
  • No gossip. Load counters are router-side local state only (v1); gossip is a future option for proactive capacity rebalancing (spike §4).
  • CPU-testable. Router logic is plain data structures — no tensors, no GPU. tch-rs is !Send; this module touches none of it.

§Determinism

HRW requires a hash that is stable across processes (so two routers in the same cell pick the same owner for the same session). DefaultHasher::new() is documented to use constant keys, so its output is deterministic across runs and processes. We do not use RandomState (which is randomly seeded per process and would break cross-router agreement).

Structs§

CellRouter
Router-side load + health state for a cell’s replica set.
InferenceServerInfo
Information about a candidate inference server in the cell.
Placement
Outcome of a placement decision.
ReplicaId
Opaque identifier for an inference replica used only for placement.
SessionAffinity
Session→owner affinity map with heartbeat-lease renewal.

Enums§

ExcludeReason
Reason a node was excluded from placement (for the failure log / tests).
PlacementPolicy
Where to place a session within the replica set.

Constants§

DEFAULT_DOWN_TTL
Default negative-cache TTL on a dial-fail (avoid hammering a down node).
DEFAULT_LEASE_DURATION
Default lease for a session→owner affinity binding. Renewed on every heartbeat; expiry forces a fresh HRW on the next request.
DEFAULT_STALL_DEADLINE
Default deadline before a token-starved stream is declared stalled.

Functions§

hrw_select
Highest Random Weight (Rendezvous Hashing) selection.
hrw_select_skip
Select the skip-th HRW choice (0 = primary, 1 = first fallback, …).
least_loaded_select
Select the least-loaded candidate (lowest active_sessions, ties broken by most free GPU memory). Returns None for an empty set.

Type Aliases§

SessionId
Identifier for a chat / inference session (used as the HRW placement key).