Verification gates — code tour

Leakage gate — episode-disjoint, family-pure committed splits

Asserts: No episode contributes windows to more than one partition, and every partition draws from its declared dataset family only.

Where: tests/integration/test_leakage.py:62 — def test_committed_splits_are_disjoint_and_family_pure(grid): (all line numbers derived at build time)

On failure: pytest fails naming the grid and the episode id that appears in two partitions (or the window whose family is wrong).

How to read it: The assertion that matters is assert_disjoint_triples() over the committed split files of each grid — the exact files every run reads. CI re-runs this test by name on every push so a leaky split can never merge silently.

Run it: pytest tests/integration/test_leakage.py -q

Splits audit — the validator is itself validated

Asserts: The split auditor detects a planted duplicate episode and a missing held-out class — proving the leakage checks can actually catch what they claim to.

Where: tests/integration/test_splits_audit.py:84 — def test_audit_flags_episode_leakage(): (all line numbers derived at build time)

On failure: pytest fails because a deliberately corrupted split was NOT flagged — i.e. the audit machinery itself regressed.

How to read it: This is the mirror of the leakage gate: it corrupts a split on purpose and asserts the auditor raises. A green run means the leakage gate's 'pass' is meaningful, not vacuous.

Run it: pytest tests/integration/test_splits_audit.py -q

Analytic floor gate — majority ≈ 1/k, learned ≥ floor

Asserts: Per (task × grid): the majority baseline's balanced accuracy equals the analytic 1/k chance floor (tolerance 0.02), and no learned baseline scores below the floor on the test split — direction-aware, pending cells skipped (never a false fail).

Where: src/evemtbench/coverage.py:379 — def check_floor( (all line numbers derived at build time)

On failure: exit 1 listing each violating (task, grid, baseline) with its value vs the floor — e.g. a mis-mapped label set or an evaluation bug pushing majority off 1/k.

How to read it: A known-answer test over the whole results matrix: the floor is computable by hand (1/k for k classes), so agreement is evidence the metric pipeline, label derivations and aggregation are wired correctly end to end.

Current report: 564 comparisons across 96 groups, 2 violations, 0 skipped (pending cells).

Run it: PYTHONPATH=src python -m evemtbench.coverage --check-floor

Freshness gate — committed views byte-match a rebuild

Asserts: RESULTS.md, leaderboard.html, audit/ and coverage_snapshot.json equal a fresh rebuild from outputs/ byte-for-byte (one volatile 'updated …' line normalized).

Where: hpc/check_results_fresh.py:268 — if _normalize(committed) != _normalize(fresh): (all line numbers derived at build time)

On failure: exit 1 with a unified diff of the stale file; the pre-commit hook blocks the commit until `--fix` regenerates it.

How to read it: The one comparison that matters is committed-vs-fresh equality per rendered file. Because the builders are pure functions of outputs/, every number on this site is mechanically traceable to a result.json — hand-editing a published number is impossible without the gate failing.

Run it: python hpc/check_results_fresh.py

Artifact integrity — the outputs corpus itself is checked

Asserts: Every done cell carries the full seed set, a schema-valid provenance record with non-null git/lockfile pins, run_record.seeds == result.seeds, and existing checkpoints; --deep additionally verifies every predictions parquet's row count against n_test (truncated-write detection).

Where: src/evemtbench/integrity.py:162 — def check_outputs( (all line numbers derived at build time)

On failure: exit 1 listing each problem as '<cell>: <what is inconsistent>'; pending cells are skipped, never failed (coverage owns those).

How to read it: CI tests validate the code; this gate validates the artifacts the sweep left behind, which only exist on the loop host. It also reports the distinct (git sha, lockfile hash) pairs across the corpus — the evidence behind the paper's 'attributable to a frozen configuration' sentence. Run at every freeze.

Run it: PYTHONPATH=src python -m evemtbench.integrity

Contract pin — SHA-256-pinned window contract and inputs

Asserts: The vendored window contract matches its pinned SHA-256 and version; every run record pins the dependency lockfile hash; every windowed input carries a sidecar manifest whose feature hash is verified at load time.

Where: src/evemtbench/contract/__init__.py:51 — def verify_pin() -> list[str]: (all line numbers derived at build time)

On failure: verify_pin() returns a non-empty problem list (version or SHA mismatch) and the contract-conformance tests fail.

How to read it: Three pins close the input side: contract SHA (window geometry can't drift), the per-file feature hash checked when windows are memory-mapped (channel order can't drift), and the environment. Read the environment pin precisely: a lockfile SHA in a run record proves only which file sat in the repo, NOT which interpreter ran — so the real check is that each record's independently captured numpy/pandas/scikit-learn/scipy/torch versions agree with the captured results environment (env/results-v<version>.lock, verified by hpc/verify_results_env.py). Two independent captures agreeing is the guarantee; the hash alone is a pointer.

Run it: pytest tests/integration/test_contract_conformance.py -q