Skip to main content

Crate hyprstream_calibration

Crate hyprstream_calibration 

Source
Expand description

§hyprstream-calibration — calibration metrics library (System One DAG node P0.2)

Calibration is the product thesis of the System One program, and this crate owns its measurement half: every number a gate (P0.5, P1.4, P2.3) or a published evaluation reports is computed here, under one pinned protocol, so results are comparable across arms, splits, and releases. Primitives, not opinions: these are pure functions over probability vectors — no model code, no fitting (post-hoc parameter fitting is P2.2), no serving policy.

§What is measured

  • Nominal calibration (metrics): top-label ECE, multiclass Brier score, negative log-likelihood, and reliability curves, all over the pinned binning (bins).
  • Ordinal calibration (metrics): class-wise SCE/ACE and the ranked probability score (RPS) for the score primitive — top-label ECE is class-blind on ordinal tasks and cannot see the tail errors that corrupt the probability-weighted value (S6b2, ORCU arXiv:2410.15658). RPS is a proper scoring rule on the cumulative distribution.
  • Selective prediction (selective): risk–coverage curves over confidence-thresholded answers; abstention (null answers in the jev-1 profile) is the data this operates on.
  • Conformal nonconformity (conformal): vanilla APS/RAPS scores and prediction sets for noul/choice, and ordinal contiguous-interval nonconformity for score — RPS-based and min-CPS-style sliding window — because probability-sorted APS/RAPS sets are order-blind and can be non-contiguous over an ordered scale (S6b2). MAPIE ships no ordinal method; these are implemented in-house. Split-conformal quantiles use the standard finite-sample correction.
  • Cross-question / joint coverage (joint): conformal coverage guarantees are marginal (per question); the fraction of batch rows in which every question is covered is reported separately, against the product-of-marginals independence reference.

§The gate measurement protocol (protocol)

Pinned so gate numerics are reproducible and comparable (consumed by P1.4 legs b/c/e and P2.3):

  • 15 equal-mass bins (protocol::GATE_BIN_COUNT) for ECE and reliability curves — equal-mass, not the classic equal-width, so bins are populated under peaked real-world confidence distributions.
  • Bootstrap percentile CIs (bootstrap) at 95% over 1000 resamples (protocol::GATE_BOOTSTRAP_RESAMPLES, protocol::GATE_CI_LEVEL); gate comparisons that pin a bound (e.g. P1.4 leg b’s “≤ 2×”) condition on the upper CI bound.
  • Per-field macro rule (protocol::macro_average): metrics are computed per question field and averaged with equal field weights — never pooled over rows, so high-volume fields cannot dilute a miscalibrated rare field.
  • Shift-split definition (protocol::ShiftSplit): the shift split is a held-out workflow family — evaluation families disjoint from every fitting/calibration family, enforced at construction.

§Disclosure (disclosure)

The pre-committed disclosure language that must accompany every published number is pinned here as data (disclosure::DISCLOSURE): claims are scoped to mechanically-checkable families plus teacher-agreement splits plus realized telemetry from a stated date, and calibration-to-teacher is not calibration-to-reality.

Re-exports§

pub use bins::Bin;
pub use bins::Binning;
pub use bootstrap::bootstrap_ci;
pub use bootstrap::ConfidenceInterval;
pub use bootstrap::SplitMix64;
pub use conformal::aps_nonconformity;
pub use conformal::aps_set;
pub use conformal::conformal_quantile;
pub use conformal::min_cps_interval;
pub use conformal::min_cps_nonconformity;
pub use conformal::ordinal_aps_interval;
pub use conformal::ordinal_set_rps;
pub use conformal::raps_nonconformity;
pub use conformal::raps_set;
pub use conformal::rps_nonconformity;
pub use error::MetricError;
pub use joint::joint_coverage;
pub use joint::JointCoverage;
pub use metrics::ace;
pub use metrics::brier;
pub use metrics::ece;
pub use metrics::expected_value;
pub use metrics::nll;
pub use metrics::reliability_curve;
pub use metrics::rps;
pub use metrics::sce;
pub use protocol::FieldMetrics;
pub use protocol::GateReport;
pub use protocol::MacroMetrics;
pub use protocol::ShiftSplit;
pub use protocol::GATE_BIN_COUNT;
pub use protocol::GATE_BOOTSTRAP_RESAMPLES;
pub use protocol::GATE_CI_LEVEL;
pub use selective::accuracy_at_coverage;
pub use selective::risk_coverage_curve;
pub use selective::selective_auc;
pub use selective::RiskCoverage;

Modules§

bins
Binning for calibration curves and error metrics.
bootstrap
Deterministic bootstrap confidence intervals for gate statistics.
conformal
Conformal nonconformity scores and prediction sets.
disclosure
Pre-committed disclosure language.
error
Error type for metric and conformal computations.
joint
Cross-question / joint coverage.
metrics
Calibration and proper-scoring metrics over probability vectors.
protocol
The pinned gate measurement protocol.
selective
Selective prediction: risk–coverage analysis over confidence-thresholded answers.