Skip to content

Commit 8a959e9

Browse files
Sync to VibeCodingScientist/VCBench@301cb48 (residual Hauser strip)
Two more references to a personal name caught by a wider grep (audits/ + tests/) — both swapped to 'VCBench (2026)'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8fd22e3 commit 8a959e9

2 files changed

Lines changed: 3 additions & 3 deletions

File tree

audits/PROGRESS_2026-05-02_team_review.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
## Executive summary
1010

11-
We have a solid foundation: the three reusable methodological tools the spec flags as the most-important architectural decision are shipped with full tests; every Eq. (1–9) of Hauser et al. 2026 is implemented in the new `src/vcbench/` package and validated against §I.4 reference values; the leak-corrected Arc State wrapper has the loud docstring + a CI guard the spec asked for. **210 / 210 unit tests pass.**
11+
We have a solid foundation: the three reusable methodological tools the spec flags as the most-important architectural decision are shipped with full tests; every Eq. (1–9) of VCBench (2026) is implemented in the new `src/vcbench/` package and validated against §I.4 reference values; the leak-corrected Arc State wrapper has the loud docstring + a CI guard the spec asked for. **210 / 210 unit tests pass.**
1212

1313
What we don't have yet: working model runtime (the wrappers currently `NotImplementedError` and delegate to the legacy `src/models/run_*.py`), a CLI entry point (`make all` shells out to legacy bash scripts), most per-dim configs, the docs site, the leaderboard / submissions infrastructure, and the VPS-only finals (HF checkpoint publish + fresh-repro validation pass).
1414

@@ -307,7 +307,7 @@ Estimated total: **~10 local sessions + ~20h VPS time** to defensible v1.0.
307307

308308
If the team wants a "safe to share with one trusted reviewer" snapshot today, the following claims hold:
309309

310-
- ✅ Every Eq. (1–9) of Hauser et al. 2026 is implemented in code, tested, and validated against the §I.4 reference values within ±0.001.
310+
- ✅ Every Eq. (1–9) of VCBench (2026) is implemented in code, tested, and validated against the §I.4 reference values within ±0.001.
311311
- ✅ The three reusable methodological tools (`vcbench.protocols.common_label_set`, `vcbench.probes.spread_error_correlation`, `vcbench.contamination`) work standalone with `pip install -e .`.
312312
- ✅ The Arc State leak — the most reputationally-sensitive issue in the manuscript — has a wrapper that defaults to the leak-corrected config, refuses to load the deprecated config without an explicit opt-in, and a CI workflow that fails the PR if the active config has nonzero train/test overlap.
313313
- ✅ The pre-registered VC Level mapping is frozen in `configs/pre_registration.yaml` with `expected_assignments` matching §I.5 exactly (only TranscriptFormer at Level 2).

tests/reference_values.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"_doc": "VCBench reference values (Hauser et al. 2026, §I.4 capability matrix + §I.3 reference test cases). Any drift in these values triggers test failure in tests/unit/test_reference_values.py. Edit only when the manuscript itself changes.",
2+
"_doc": "VCBench reference values (VCBench (2026), §I.4 capability matrix + §I.3 reference test cases). Any drift in these values triggers test failure in tests/unit/test_reference_values.py. Edit only when the manuscript itself changes.",
33
"_tolerance_default": 0.001,
44

55
"dim_a": {

0 commit comments

Comments
 (0)