hyprstream_calibration/lib.rs
1//! # hyprstream-calibration — calibration metrics library (System One DAG node P0.2)
2//!
3//! Calibration is the product thesis of the System One program, and this crate owns its
4//! *measurement* half: every number a gate (P0.5, P1.4, P2.3) or a published evaluation
5//! reports is computed here, under one pinned protocol, so results are comparable across
6//! arms, splits, and releases. Primitives, not opinions: these are pure functions over
7//! probability vectors — no model code, no fitting (post-hoc parameter fitting is P2.2),
8//! no serving policy.
9//!
10//! ## What is measured
11//!
12//! - **Nominal calibration** ([`metrics`]): top-label ECE, multiclass Brier score,
13//! negative log-likelihood, and reliability curves, all over the pinned binning
14//! ([`bins`]).
15//! - **Ordinal calibration** ([`metrics`]): class-wise SCE/ACE and the ranked probability
16//! score (RPS) for the `score` primitive — top-label ECE is class-blind on ordinal
17//! tasks and cannot see the tail errors that corrupt the probability-weighted value
18//! (S6b2, ORCU arXiv:2410.15658). RPS is a proper scoring rule on the *cumulative*
19//! distribution.
20//! - **Selective prediction** ([`selective`]): risk–coverage curves over
21//! confidence-thresholded answers; abstention (`null` answers in the jev-1 profile) is
22//! the data this operates on.
23//! - **Conformal nonconformity** ([`conformal`]): vanilla APS/RAPS scores and prediction
24//! sets for `noul`/`choice`, and ordinal contiguous-interval nonconformity for `score`
25//! — RPS-based and min-CPS-style sliding window — because probability-sorted APS/RAPS
26//! sets are order-blind and can be non-contiguous over an ordered scale (S6b2). MAPIE
27//! ships no ordinal method; these are implemented in-house. Split-conformal quantiles
28//! use the standard finite-sample correction.
29//! - **Cross-question / joint coverage** ([`joint`]): conformal coverage guarantees are
30//! *marginal* (per question); the fraction of batch rows in which *every* question is
31//! covered is reported separately, against the product-of-marginals independence
32//! reference.
33//!
34//! ## The gate measurement protocol ([`protocol`])
35//!
36//! Pinned so gate numerics are reproducible and comparable (consumed by P1.4 legs b/c/e
37//! and P2.3):
38//!
39//! - **15 equal-mass bins** ([`protocol::GATE_BIN_COUNT`]) for ECE and reliability
40//! curves — equal-mass, not the classic equal-width, so bins are populated under
41//! peaked real-world confidence distributions.
42//! - **Bootstrap percentile CIs** ([`bootstrap`]) at 95% over 1000 resamples
43//! ([`protocol::GATE_BOOTSTRAP_RESAMPLES`], [`protocol::GATE_CI_LEVEL`]); gate
44//! comparisons that pin a bound (e.g. P1.4 leg b's "≤ 2×") condition on the **upper CI
45//! bound**.
46//! - **Per-field macro rule** ([`protocol::macro_average`]): metrics are computed per
47//! question field and averaged with equal field weights — never pooled over rows, so
48//! high-volume fields cannot dilute a miscalibrated rare field.
49//! - **Shift-split definition** ([`protocol::ShiftSplit`]): the shift split is a
50//! held-out workflow *family* — evaluation families disjoint from every
51//! fitting/calibration family, enforced at construction.
52//!
53//! ## Disclosure ([`disclosure`])
54//!
55//! The pre-committed disclosure language that must accompany every published number is
56//! pinned here as data ([`disclosure::DISCLOSURE`]): claims are scoped to
57//! mechanically-checkable families plus teacher-agreement splits plus realized telemetry
58//! from a stated date, and calibration-to-teacher is not calibration-to-reality.
59
60pub mod bins;
61pub mod bootstrap;
62pub mod conformal;
63pub mod disclosure;
64pub mod error;
65pub mod joint;
66pub mod metrics;
67pub mod protocol;
68pub mod selective;
69
70pub use bins::{Bin, Binning};
71pub use bootstrap::{bootstrap_ci, ConfidenceInterval, SplitMix64};
72pub use conformal::{
73 aps_nonconformity, aps_set, conformal_quantile, min_cps_interval, min_cps_nonconformity,
74 ordinal_aps_interval, ordinal_set_rps, raps_nonconformity, raps_set, rps_nonconformity,
75};
76pub use error::MetricError;
77pub use joint::{joint_coverage, JointCoverage};
78pub use metrics::{ace, brier, ece, expected_value, nll, reliability_curve, rps, sce};
79pub use protocol::{
80 FieldMetrics, GateReport, MacroMetrics, ShiftSplit, GATE_BIN_COUNT, GATE_BOOTSTRAP_RESAMPLES,
81 GATE_CI_LEVEL,
82};
83pub use selective::{accuracy_at_coverage, risk_coverage_curve, selective_auc, RiskCoverage};