Skip to main content

hyprstream_calibration/
lib.rs

1//! # hyprstream-calibration — calibration metrics library (System One DAG node P0.2)
2//!
3//! Calibration is the product thesis of the System One program, and this crate owns its
4//! *measurement* half: every number a gate (P0.5, P1.4, P2.3) or a published evaluation
5//! reports is computed here, under one pinned protocol, so results are comparable across
6//! arms, splits, and releases. Primitives, not opinions: these are pure functions over
7//! probability vectors — no model code, no fitting (post-hoc parameter fitting is P2.2),
8//! no serving policy.
9//!
10//! ## What is measured
11//!
12//! - **Nominal calibration** ([`metrics`]): top-label ECE, multiclass Brier score,
13//!   negative log-likelihood, and reliability curves, all over the pinned binning
14//!   ([`bins`]).
15//! - **Ordinal calibration** ([`metrics`]): class-wise SCE/ACE and the ranked probability
16//!   score (RPS) for the `score` primitive — top-label ECE is class-blind on ordinal
17//!   tasks and cannot see the tail errors that corrupt the probability-weighted value
18//!   (S6b2, ORCU arXiv:2410.15658). RPS is a proper scoring rule on the *cumulative*
19//!   distribution.
20//! - **Selective prediction** ([`selective`]): risk–coverage curves over
21//!   confidence-thresholded answers; abstention (`null` answers in the jev-1 profile) is
22//!   the data this operates on.
23//! - **Conformal nonconformity** ([`conformal`]): vanilla APS/RAPS scores and prediction
24//!   sets for `noul`/`choice`, and ordinal contiguous-interval nonconformity for `score`
25//!   — RPS-based and min-CPS-style sliding window — because probability-sorted APS/RAPS
26//!   sets are order-blind and can be non-contiguous over an ordered scale (S6b2). MAPIE
27//!   ships no ordinal method; these are implemented in-house. Split-conformal quantiles
28//!   use the standard finite-sample correction.
29//! - **Cross-question / joint coverage** ([`joint`]): conformal coverage guarantees are
30//!   *marginal* (per question); the fraction of batch rows in which *every* question is
31//!   covered is reported separately, against the product-of-marginals independence
32//!   reference.
33//!
34//! ## The gate measurement protocol ([`protocol`])
35//!
36//! Pinned so gate numerics are reproducible and comparable (consumed by P1.4 legs b/c/e
37//! and P2.3):
38//!
39//! - **15 equal-mass bins** ([`protocol::GATE_BIN_COUNT`]) for ECE and reliability
40//!   curves — equal-mass, not the classic equal-width, so bins are populated under
41//!   peaked real-world confidence distributions.
42//! - **Bootstrap percentile CIs** ([`bootstrap`]) at 95% over 1000 resamples
43//!   ([`protocol::GATE_BOOTSTRAP_RESAMPLES`], [`protocol::GATE_CI_LEVEL`]); gate
44//!   comparisons that pin a bound (e.g. P1.4 leg b's "≤ 2×") condition on the **upper CI
45//!   bound**.
46//! - **Per-field macro rule** ([`protocol::macro_average`]): metrics are computed per
47//!   question field and averaged with equal field weights — never pooled over rows, so
48//!   high-volume fields cannot dilute a miscalibrated rare field.
49//! - **Shift-split definition** ([`protocol::ShiftSplit`]): the shift split is a
50//!   held-out workflow *family* — evaluation families disjoint from every
51//!   fitting/calibration family, enforced at construction.
52//!
53//! ## Disclosure ([`disclosure`])
54//!
55//! The pre-committed disclosure language that must accompany every published number is
56//! pinned here as data ([`disclosure::DISCLOSURE`]): claims are scoped to
57//! mechanically-checkable families plus teacher-agreement splits plus realized telemetry
58//! from a stated date, and calibration-to-teacher is not calibration-to-reality.
59
60pub mod bins;
61pub mod bootstrap;
62pub mod conformal;
63pub mod disclosure;
64pub mod error;
65pub mod joint;
66pub mod metrics;
67pub mod protocol;
68pub mod selective;
69
70pub use bins::{Bin, Binning};
71pub use bootstrap::{bootstrap_ci, ConfidenceInterval, SplitMix64};
72pub use conformal::{
73    aps_nonconformity, aps_set, conformal_quantile, min_cps_interval, min_cps_nonconformity,
74    ordinal_aps_interval, ordinal_set_rps, raps_nonconformity, raps_set, rps_nonconformity,
75};
76pub use error::MetricError;
77pub use joint::{joint_coverage, JointCoverage};
78pub use metrics::{ace, brier, ece, expected_value, nll, reliability_curve, rps, sce};
79pub use protocol::{
80    FieldMetrics, GateReport, MacroMetrics, ShiftSplit, GATE_BIN_COUNT, GATE_BOOTSTRAP_RESAMPLES,
81    GATE_CI_LEVEL,
82};
83pub use selective::{accuracy_at_coverage, risk_coverage_curve, selective_auc, RiskCoverage};