Skip to main content

Module metrics

Module metrics 

Source
Expand description

Calibration and proper-scoring metrics over probability vectors.

All functions take parallel slices: one probability distribution per observation plus the realized label index. A noul answer is the 2-wide distribution [P(false), P(true)]; choice and score distributions are as wide as their option/level cardinality. Width may vary across observations (score levels are runtime-declared), except where a class-wise metric requires a shared class axis (sce, ace).

  • ECE (ece) — top-label, over the pinned equal-mass protocol binning.
  • Brier (brier) — multiclass Brier score Σₖ (pₖ − yₖ)².
  • NLL (nll) — negative log-likelihood of the realized label.
  • RPS (rps) — ranked probability score on the cumulative distribution, normalized by K − 1; the proper ordinal metric for score (S6b2).
  • SCE / ACE (sce, ace) — class-wise calibration error, static (equal-width) vs adaptive (equal-mass) binning per Nixon et al. 2019; the metrics that see ordinal tail miscalibration which top-label ECE misses (ORCU, arXiv:2410.15658).
  • Expected value (expected_value) — the probability-weighted score readout Σₖ k·pₖ, exposed so re-fits can be validated against weighted-value error (S6b2 value guardrail: temperature moves the expectation without moving the mode).

Functions§

ace
Adaptive calibration error — class-wise, equal-mass bins (Nixon et al. 2019).
argmax
Argmax index with earliest-index tie-breaking (jev-1 profile decision D6).
brier
Mean multiclass Brier score Σₖ (pₖ − yₖ)² (lower is better; 0 is perfect).
ece
Top-label ECE under the pinned protocol binning (n_bins equal-mass bins).
ece_from_bins
Top-label expected calibration error over binning.
expected_value
Probability-weighted value of an ordinal distribution over level indices: E = Σₖ k·pₖ (level values default to their indices per the jev-1 profile).
nll
Mean negative log-likelihood of the realized label.
reliability_curve
Reliability curve over the top-label confidence axis.
rps
Mean ranked probability score over the cumulative distribution, normalized by K − 1 so scores are comparable across runtime-declared level cardinalities: RPS = Σₖ₌₀..K₋₂ (Fₖ − 1{y ≤ k})² / (K − 1) (lower is better; 0 is perfect).
rps_single
RPS of a single observation (normalized by K − 1; K = 1 scores 0).
sce
Static calibration error — class-wise, equal-width bins (Nixon et al. 2019).