Skip to main content

Module protocol

Module protocol 

Source
Expand description

The pinned gate measurement protocol.

Every gate that consumes calibration numbers (P0.5 baselines, P1.4 legs b/c/e, P2.3) measures them this way. The constants are the contract; GateReport is the serializable artifact a gate run emits. Nothing here fits parameters — fitting is P2.2; this module defines measurement only.

§The protocol, pinned

  1. Binning: GATE_BIN_COUNT equal-mass bins for ECE and reliability curves (crate::bins::Binning::equal_mass). Equal-mass, not classic equal-width, so bins stay populated under peaked confidence distributions.
  2. Uncertainty: percentile bootstrap CIs at GATE_CI_LEVEL over GATE_BOOTSTRAP_RESAMPLES resamples, seeded and recorded. Gate comparisons that pin a bound (P1.4 leg b’s “macro-ECE gap ≤ 2×”) condition on the upper CI bound, never the point estimate.
  3. Aggregation — per-field macro rule: metrics are computed per question field and averaged with equal field weight (macro_average). Rows are never pooled across fields: a high-volume field must not dilute a miscalibrated rare one.
  4. Shift split (ShiftSplit): the shift evaluation split is a held-out workflow family — evaluation families disjoint from every family used for fitting or calibration. Item-level disjointness is not the shift split.
  5. Per-primitive metrics (P1.4 leg c): ECE for noul/choice; SCE/ACE + RPS for score (top-label ECE is class-blind on ordinal tasks — S6b2).
  6. Joint coverage: conformal coverage is marginal; cross-question joint coverage is reported separately (crate::joint).

Structs§

FieldMetrics
One question field’s measured metrics (one row of the macro rule).
GateReport
A gate run’s serializable measurement artifact.
MacroMetrics
Macro-averaged metric with its bootstrap CI, as reported by a gate.
ShiftSplit
The shift-split definition: which workflow families may be used for fitting/calibration and which are held out for evaluation.

Constants§

GATE_BIN_COUNT
Pinned bin count for ECE and reliability curves (equal-mass).
GATE_BOOTSTRAP_RESAMPLES
Pinned bootstrap resample count for gate confidence intervals.
GATE_CI_LEVEL
Pinned confidence level for gate intervals.

Functions§

macro_average
Per-field macro aggregation: unweighted mean of per-field metric values. Returns None for an empty field set (a report with no fields is malformed, not 0).