Expand description
Calibration and proper-scoring metrics over probability vectors.
All functions take parallel slices: one probability distribution per observation plus
the realized label index. A noul answer is the 2-wide distribution
[P(false), P(true)]; choice and score distributions are as wide as their
option/level cardinality. Width may vary across observations (score levels are
runtime-declared), except where a class-wise metric requires a shared class axis
(sce, ace).
- ECE (
ece) — top-label, over the pinned equal-mass protocol binning. - Brier (
brier) — multiclass Brier scoreΣₖ (pₖ − yₖ)². - NLL (
nll) — negative log-likelihood of the realized label. - RPS (
rps) — ranked probability score on the cumulative distribution, normalized byK − 1; the proper ordinal metric forscore(S6b2). - SCE / ACE (
sce,ace) — class-wise calibration error, static (equal-width) vs adaptive (equal-mass) binning per Nixon et al. 2019; the metrics that see ordinal tail miscalibration which top-label ECE misses (ORCU, arXiv:2410.15658). - Expected value (
expected_value) — the probability-weighted score readoutΣₖ k·pₖ, exposed so re-fits can be validated against weighted-value error (S6b2 value guardrail: temperature moves the expectation without moving the mode).
Functions§
- ace
- Adaptive calibration error — class-wise, equal-mass bins (Nixon et al. 2019).
- argmax
- Argmax index with earliest-index tie-breaking (jev-1 profile decision D6).
- brier
- Mean multiclass Brier score
Σₖ (pₖ − yₖ)²(lower is better; 0 is perfect). - ece
- Top-label ECE under the pinned protocol binning (
n_binsequal-mass bins). - ece_
from_ bins - Top-label expected calibration error over
binning. - expected_
value - Probability-weighted value of an ordinal distribution over level indices:
E = Σₖ k·pₖ(level values default to their indices per the jev-1 profile). - nll
- Mean negative log-likelihood of the realized label.
- reliability_
curve - Reliability curve over the top-label confidence axis.
- rps
- Mean ranked probability score over the cumulative distribution, normalized by
K − 1so scores are comparable across runtime-declared level cardinalities:RPS = Σₖ₌₀..K₋₂ (Fₖ − 1{y ≤ k})² / (K − 1)(lower is better; 0 is perfect). - rps_
single - RPS of a single observation (normalized by
K − 1;K = 1scores 0). - sce
- Static calibration error — class-wise, equal-width bins (Nixon et al. 2019).