Expand description
Selective prediction: risk–coverage analysis over confidence-thresholded answers.
Abstention is a first-class answer value in the jev-1 profile (null = abstained),
and selective accuracy is the metric that prices it: accuracy among the answers a
threshold policy would have kept, as a function of coverage. The curve is computed
exactly (every distinct threshold), not sampled.
Structs§
- Risk
Coverage - One point on the risk–coverage curve.
Functions§
- accuracy_
at_ coverage - Accuracy at the largest coverage
<= target_coverage(step semantics: a threshold policy realizes only the curve’s own points). ReturnsNoneif the target is below the smallest realized coverage. - risk_
coverage_ curve - Exact risk–coverage curve: sort by confidence descending; at each distinct threshold, coverage = kept fraction and accuracy = mean correctness among kept.
- selective_
auc - Area under the risk–coverage curve (coverage on x, accuracy on y), trapezoidal over the realized points with the left edge extended to coverage 0 at the first point’s accuracy. Higher is better; 1.0 is an oracle ranker.