Expand description
The pinned gate measurement protocol.
Every gate that consumes calibration numbers (P0.5 baselines, P1.4 legs b/c/e, P2.3)
measures them this way. The constants are the contract; GateReport is the
serializable artifact a gate run emits. Nothing here fits parameters — fitting is
P2.2; this module defines measurement only.
§The protocol, pinned
- Binning:
GATE_BIN_COUNTequal-mass bins for ECE and reliability curves (crate::bins::Binning::equal_mass). Equal-mass, not classic equal-width, so bins stay populated under peaked confidence distributions. - Uncertainty: percentile bootstrap CIs at
GATE_CI_LEVELoverGATE_BOOTSTRAP_RESAMPLESresamples, seeded and recorded. Gate comparisons that pin a bound (P1.4 leg b’s “macro-ECE gap ≤ 2×”) condition on the upper CI bound, never the point estimate. - Aggregation — per-field macro rule: metrics are computed per question field
and averaged with equal field weight (
macro_average). Rows are never pooled across fields: a high-volume field must not dilute a miscalibrated rare one. - Shift split (
ShiftSplit): the shift evaluation split is a held-out workflow family — evaluation families disjoint from every family used for fitting or calibration. Item-level disjointness is not the shift split. - Per-primitive metrics (P1.4 leg c): ECE for
noul/choice; SCE/ACE + RPS forscore(top-label ECE is class-blind on ordinal tasks — S6b2). - Joint coverage: conformal coverage is marginal; cross-question joint coverage
is reported separately (
crate::joint).
Structs§
- Field
Metrics - One question field’s measured metrics (one row of the macro rule).
- Gate
Report - A gate run’s serializable measurement artifact.
- Macro
Metrics - Macro-averaged metric with its bootstrap CI, as reported by a gate.
- Shift
Split - The shift-split definition: which workflow families may be used for fitting/calibration and which are held out for evaluation.
Constants§
- GATE_
BIN_ COUNT - Pinned bin count for ECE and reliability curves (equal-mass).
- GATE_
BOOTSTRAP_ RESAMPLES - Pinned bootstrap resample count for gate confidence intervals.
- GATE_
CI_ LEVEL - Pinned confidence level for gate intervals.
Functions§
- macro_
average - Per-field macro aggregation: unweighted mean of per-field metric values.
Returns
Nonefor an empty field set (a report with no fields is malformed, not 0).