Expand description
§hyprstream-bench — verifiable-outcome calibration benchmark (System One, P0.6)
The program’s truth anchor: score-bearing items with mechanically
verifiable outcomes, emitted as jev-1 question specs
([hyprstream_decision]). Everything is procedurally generated fresh
from published generators and pinned seeds (gen) — no public corpora
in the gating strata, so the benchmark is immune to teacher-pretraining
contamination by construction.
§What the release pins
- Generators + seeds (
gen,rng::BenchRng— a frozen splitmix64 stream). Same config ⇒ bit-identical items, forever. The PRNG, the canonical item bytes (Item::canonical_bytes), and the family taxonomy are all part of the frozen contract. - Adversarial strata (
Stratum):clean,nearmiss(near-miss JSON, near-miss arithmetic, close distractors), reported per stratum by the P0.2 measurement protocol. - Cyclic-permutation stratum (CircularEval precedent, S6b1): every
generated choice item is additionally emitted under all
k−1non- identity cyclic option rotations, linked byItem::group, truth rotating with the options — so option-order robustness (flip rate, ECE drift) is measured at zero retraining cost. Score levels are not permuted: level order is the semantics there. Noul has no options. - Family-level holdout firewall (
Family::designation,Manifest::gate_families):temporalandsyllogismare designated gate families — never synthesized into training data; the P1.4 leg-(e) zero-shot transfer gate measures the model on exactly these. - Frozen item manifest (
manifest,manifest/vob-1.0.manifest.json): per-item blake3 hashes, frozen before any P1.3 synthesis run. P1.3 consumes the manifest as its item-level contamination firewall and its family firewall list.
§Licensing
Harness: Apache-2.0. Generated items: CC-BY-4.0 (the default public
release shape per the program licensing policy). See DISCLOSURE.md for
the pre-committed disclosure language that accompanies every published
result.
Re-exports§
pub use family::Designation;pub use family::Family;pub use family::Stratum;pub use gen::generate_all;pub use gen::BenchConfig;pub use gen::BASE_STRATA;pub use gen::DEFAULT_SEEDS_PER_STRATUM;pub use gen::SEED_BASE;pub use item::Item;pub use manifest::Manifest;pub use manifest::ManifestItem;pub use manifest::VerifyError;pub use manifest::RELEASE;
Modules§
- family
- Question families, holdout designations, and difficulty strata.
- gen
- Per-family item generators. Every generator is a pure function of
(seed, stratum)over the pinnedBenchRngstream — published generators + seeds make every item reproducible and immune to teacher-pretraining contamination. - item
- One benchmark item: a jev-1 question spec, its state, and the verifiable truth — plus the canonical byte form the frozen manifest hashes.
- manifest
- The frozen item manifest: per-item blake3 hashes, family designations, generation config, and license/disclosure pointers.
- rng
- The pinned benchmark PRNG.