Skip to content

odinfree/storm-ecdsa-harness / skills/evaluator-scoring-surface-audit

evaluator-scoring-surface-audit

Verified

Read a benchmark's evaluator, verifier and scoring source line by line for gaps between what is counted, what is checked and what is only assumed, then turn each gap into a priced lever, a tool, or a report to the organisers. Use when joining a benchmark, after any evaluator change, or when the leaderboard moves in a way your cost model does not explain.

FileKindSize
SKILL.mdskill md6.6 KB
assets/surface-table.mdmarkdown1.4 KB

assets/surface-table.md

View on GitHub
# Scoring-surface audit: <benchmark>

Evaluator commit: `<sha>`; verifier commit: `<sha>`; official runner script: `<path>`; date: `<yyyy-mm-dd>`

Score formula as coded: `<formula>` at `<file:line>`

## 1. Counted quantities

| # | quantity | file:line of the increment | increments when | unit | emitted or executed | notes |
|---|---|---|---|---|---|---|
| Q1 | | | | | | |
| Q2 | | | | | | |

## 2. Checks

| # | property checked | file:line | bound | sampling (all / sample of N / none) | on failure |
|---|---|---|---|---|---|
| C1 | | | | | |
| C2 | | | | | |

## 3. Written rules

| # | rule (quote) | enforced at file:line, or "not enforced" |
|---|---|---|
| R1 | | |

## 4. Randomness

| draw | seeded by | who can influence the seed | file:line |
|---|---|---|---|
| | | | |

## 5. Cross: gaps

| # | counted quantity or check | observation | class (exact lever / sampled lever / harness defect / intent conflict) | price (measured, with date) | action |
|---|---|---|---|---|---|
| G1 | | | | | |
| G2 | | | | | |

No gap found for: `<list the Q and C rows crossed with no finding>`

## 6. Second evaluator path

| item | local command | official runner |
|---|---|---|
| binary and flags | | |
| limits (time, memory, size) | | |
| dependency versions | | |
| differences found | | |

## 7. Reports sent

| date | to | gap | patch or test attached | reply |
|---|---|---|---|---|
| | | | | |
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall