odinfree/storm-ecdsa-harness / skills/exact-evaluator-replica
exact-evaluator-replica
VerifiedBuild and validate a fast, exact local copy of a benchmark's official evaluator: same inputs, same counts, same score, proven by a differential fuzz test and a reproduced graded log. Use when the official judge is too slow, too memory-hungry or too opaque to sit inside the edit loop of an optimization campaign (circuits, provers, gas, proof size), when many candidates must be scored per day, or when a failure-rate estimate needs fresh samples the judge cannot give.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| assets/determinism-map.md | markdown | 1.5 KB |
| scripts/difftest.py | script | 7.5 KB |
| scripts/gensynth.py | script | 6.3 KB |
| scripts/toy_bench.py | script | 13.7 KB |
assets/determinism-map.md
View on GitHub# Determinism map: <benchmark>, evaluator commit <sha>
Fill one row per random draw the evaluator makes, in the order it makes them. The map is complete when the last
section can be answered for any unit of work.
## Seed
| item | value | file:line |
|---|---|---|
| Hash or generator | <e.g. SHAKE256, one stream> | |
| Domain tag absorbed first | | |
| What is absorbed (artefact bytes, length, salt, external seed) | | |
| Serialisation of one record as absorbed | | |
## Draws
| # | draw | source stream | order | bytes per draw | amount depends on execution? | offset formula | file:line |
|---|---|---|---|---|---|---|---|
| 1 | test inputs | | first | | no | 0 .. | |
| 2 | <e.g. measurement outcomes> | | per batch, per op, in op order | | yes / no | | |
| 3 | | | | | | | |
## Unit of work
| item | value |
|---|---|
| Unit (sample, batch of samples, transaction, proof) | |
| Samples per unit, and size of the last partial unit | |
| Offset of unit b's randomness | |
| State carried from unit b to unit b+1 (must be none for parallel units) | |
## Verdict
- [ ] Every draw has an offset that does not depend on what executed. An exact parallel replica is available.
- [ ] At least one draw depends on execution: <which>. Exact parallel replica not available; serial compact replica
or calibrated proxy instead.
## Files whose change invalidates the replica
| file | last change (commit, date) |
|---|---|
| | |
## Golden logs
| artefact (hash or submission id) | judge value | replica value | replica code version | date |
|---|---|---|---|---|
| | | | | |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |