Skip to main content

hyprstream_decision/
lib.rs

1//! # hyprstream-decision — jev-1 decision/question schema IR, authoring, and Arrow emission
2//!
3//! System One program, DAG node **P0.1a** (Wave 0, critical path). This crate owns the
4//! contract half of the decision-model program: the intermediate representation for
5//! runtime-declared typed questions, a language-agnostic authoring format (YAML/JSON),
6//! the pinned canonical serialization consumed by the question encoder (P1.1), and the
7//! Arrow schema emission consumed by the ADBC/Flight SQL operator (P3.5) and the stub
8//! service (P0.7). It contains **no model code and no opinions about serving policy** —
9//! primitives, not opinions.
10//!
11//! ## jev-1 profile (clean-room)
12//!
13//! The question/answer semantics implement the *jev-1 compatibility profile*: a clean-room
14//! re-expression of the published wire shape (S6a spike: unauthenticated OpenAPI snapshot
15//! `sha256:a191f8a7df6bd6fe…` pinned 2026-09-18, cross-anchored against the MIT SDKs and
16//! the MIT adapter — all facts re-expressed here in our own prose). Three question
17//! primitives:
18//!
19//! - **noul** — a statement judged against the state, answered with `P(true) ∈ [0,1]`.
20//! - **choice** — one option chosen from 2–255 pre-enumerated options, each with an
21//!   optional rubric; answered with a probability per option.
22//! - **score** — an ordered rubric of levels (index 0-based); answered with a probability
23//!   per level. The probability-weighted value `Σᵢ i·pᵢ` is derivable
24//!   ([`confidence::expected_score`]) and is *not* stored — the raw distribution is the
25//!   trust contract.
26//!
27//! Profile v2 question types (`span`, `derived`) are **reserved** in [`QuestionKind`]:
28//! the authoring layer recognizes the tags and rejects them with a structured error, so a
29//! v2 document fails loudly instead of being silently misread as something else.
30//!
31//! ## Profile decisions (S6a ambiguity register, the six P0.1a resolutions)
32//!
33//! The upstream wire format has registered ambiguities (S6a §5). This crate pins six of
34//! them as normative profile decisions; each is enforced here:
35//!
36//! - **D1 (A2) — score requires ≥ 2 levels.** A single-level score is degenerate
37//!   (constant value, vacuous confidence). Rejected at authoring.
38//! - **D2 (A5) — choice requires ≥ 2 options; ≤ 255 options (A4) is the pinned profile
39//!   limit.** Above 255 options the two-stage recipe (score candidates, then choose among
40//!   top scorers — a client-side pattern, not a wire feature) is mandatory; see the
41//!   serialization budget below.
42//! - **D3 (A6) — `null` score levels are allowed and mean "undescribed level"**,
43//!   consistent with `null` rubrics on choice options. The level index is its identity.
44//! - **D4 (A7) — `instructions` is optional and nullable** on every question type
45//!   (the machine-readable schema trumps prose renderings).
46//! - **D5 (A8) — probability-sum tolerance:** producers MUST emit distributions with
47//!   `|Σp − 1| ≤ 1e-6` ([`confidence::PRODUCER_SUM_TOLERANCE`]); consumers MUST accept
48//!   `|Σp − 1| ≤ 1e-2` ([`confidence::CONSUMER_SUM_TOLERANCE`]). Batch construction in
49//!   this crate enforces the producer bound.
50//! - **D6 (A9) — argmax ties break toward the earliest option in authoring (insertion)
51//!   order.** Question-spec order is canonical and is preserved from the authored
52//!   document through the IR and into Arrow field metadata.
53//!
54//! Unresolved-by-design register entries: confidence-formula identity and the 255/10
55//! enforcement behavior are live-check items for P0.5/P0.7 (A1/A3/A4); non-422 error body
56//! shapes, the `usage` tokenizer identity, and request fan-out limits are profile
57//! footnotes (A10–A12) — out of scope for a schema crate.
58//!
59//! ## Abstention
60//!
61//! `null` is the abstention value everywhere: a [`answer::QuestionAnswer`] with
62//! `value: None` is an *abstained* answer; in Arrow emission the probabilities and label
63//! columns for that question are null on that row. Abstention is data, not an error —
64//! selective prediction is built on it downstream.
65//!
66//! ## Version triple
67//!
68//! Every emitted batch carries the batch-level version triple as three columns:
69//! `schema_version` (question-set schema), `model_version` (resolved model id — alias
70//! resolution is part of the serving contract), and `calib_version` (nullable; null =
71//! uncalibrated raw distribution). Calibration *parameters* are never in the schema —
72//! they live as rows in the P0.3 metrics tables.
73//!
74//! ## Canonical serialization contract (pinned per S6b1)
75//!
76//! [`serialize`] is the single frozen text form fed to the encoder. Learned special
77//! **anchor token placeholder immediately AFTER each option's rubric text** (option →
78//! anchor, never a trailing anchor section): marker-adjacent readout is the proven
79//! pattern, and our backbone's 18/24 causal linear-attention layers cannot bind a
80//! trailing anchor section to its options across 255 positions. Format changes swing
81//! frozen-LLM accuracy by up to 76 pp (Sclar et al.), so there is exactly ONE canonical
82//! serialization and it does not change without a schema version bump. Question ids are
83//! caller-facing only and are never serialized to the model. Noul questions carry
84//! **exactly one anchor** (not one per outcome): with no per-option rubric blocks, the
85//! single anchor follows the optional true/false criteria blocks and is the position
86//! where the sigmoid readout is taken (P1.1) — the same adjacent-after principle,
87//! applied to the statement's rubric blocks.
88//!
89//! **Acceptance criterion — token budget:** a serialized request (state + all questions)
90//! must fit the **2048-token budget at 2k ctx** ([`serialize::MAX_SERIALIZED_SPEC_TOKENS`]).
91//! When a question cannot fit its options within the budget — the canonical case being
92//! cardinality above 255 — the two-stage score-then-choice recipe becomes mandatory.
93//! Token counting is tokenizer-bound (P1.1); this crate exposes the byte length and the
94//! budget constant as the contract surface.
95//!
96//! ## Confidence-from-peakedness (derived, normative formulas)
97//!
98//! Raw distributions are primary. Jev-style confidence is *derivable* from a distribution
99//! via the normative S6a formulas ([`confidence`]):
100//!
101//! - choice: `(p_max − 1/n) / (1 − 1/n)` (n = 1 → 1.0)
102//! - score: `max(0, 1 − MAD_mode / MAD_uniform)`
103//!
104//! These exist so ecosystem thresholds port; they are conveniences, not stored contract.
105//!
106//! ## Arrow emission contract
107//!
108//! [`arrow::DecisionSchema`] emits, per question, a nullable
109//! `fixed_size_list<f32>` **probabilities** column (width = cardinality; noul is the
110//! 2-wide `[P(false), P(true)]` distribution), a nullable utf8 **label** column
111//! (null = abstained), and — while v1 reserves it — a nullable `list<utf8>`
112//! **conformal prediction-set** column that may be entirely unpopulated. The label list
113//! for a question lives in the probabilities column's **Arrow field metadata**
114//! (`jev.labels`, `jev.kind`) — **never in schema metadata**. Label-set evolution is
115//! schema evolution: [`arrow::DecisionSchema::check_evolution`] fails loudly when an
116//! existing question's kind or label set changes, or when conformal-set emission flips
117//! (the emitted column set changes either way); evolving either means minting a
118//! new schema version.
119
120pub mod answer;
121pub mod arrow;
122pub mod author;
123pub mod confidence;
124pub mod entry;
125pub mod error;
126pub mod serialize;
127pub mod spec;
128
129pub use answer::{AnswerValue, QuestionAnswer, VersionTriple};
130pub use arrow::{
131    AnswerRow, ArrowSchemaError, BatchError, DecisionSchema, EvolutionError, EvolutionReport,
132    LabelChange, LabelEvolutionError,
133};
134pub use author::{parse_json, parse_yaml, MAX_CHOICE_OPTIONS};
135pub use confidence::DistributionError;
136pub use entry::Entry;
137pub use error::{ErrorPath, SpecError, SpecErrorKind};
138pub use spec::{ChoiceOption, NoulCriteria, QuestionBody, QuestionKind, QuestionSet, QuestionSpec};