Expand description
Per-family item generators. Every generator is a pure function of
(seed, stratum) over the pinned BenchRng stream — published
generators + seeds make every item reproducible and immune to
teacher-pretraining contamination.
Modules§
- arith
- Arithmetic word problems. Truth is computed in
i64; distractors and near-miss claims are the classic slips (±1, off-by-ten, transposed digits, add/subtract confusion). - jsoncheck
- JSON well-formedness / extraction. The
NearMissstratum is near-miss JSON: documents broken by exactly one classic syntax error (trailing comma, unquoted key, single quotes, missing closing brace). Validity is decided byserde_json, so truth is mechanical. - syllogism
- Set-logic syllogisms — held-out gate family (firewalled from training synthesis; measured by the P1.4 leg-(e) zero-shot transfer gate). Chains of All premises over synthetic category names; validity is decided by the chain structure, so truth is mechanical.
- temporal
- Temporal reasoning — held-out gate family (firewalled from training synthesis; measured by the P1.4 leg-(e) zero-shot transfer gate). Date arithmetic uses Hinnant’s days-from-civil algorithm implemented in-crate so truth stays mechanical and dependency-free.
- unitconv
- Unit conversion (km↔mi, kg↔lb, °C↔°F). Truth is computed with pinned conversion constants and rounded to integers so outcomes are exact.
Structs§
- Bench
Config - Benchmark generation configuration. The frozen release pins
DEFAULT_SEEDS_PER_STRATUMandSEED_BASE; regenerating with the same values must reproduce the committed manifest bit-for-bit.
Constants§
- BASE_
STRATA - The base strata every family emits (permutation expansion is mechanical and applied afterwards to choice items only).
- DEFAULT_
SEEDS_ PER_ STRATUM - Seeds per (family, base stratum, kind) in the frozen vob-1.0 release.
- SEED_
BASE - Seed base for the frozen vob-1.0 release.
Functions§
- generate_
all - Generate the full benchmark: all families × base strata × kinds × seeds, plus the cyclic-permutation expansion of every choice item.