ZenUml/conf-app / .claude/skills/ai-repair-benchmark
ai-repair-benchmark
VerifiedCompare how OpenRouter models repair the same damaged DSL; run only L1 raw-model and L2 Diagramly Prompt trials, measuring repair quality, latency, tokens, and actual cost. Use ai-repair-case-validation to investigate and regress specific cases.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.6 KB |
| agents/openai.yaml | config | 275 B |
| references/methodology.md | markdown | 5.6 KB |
| scripts/build_diagramly_prompt.tsexecutable | script | 4.3 KB |
| scripts/fixtures/smoke-candidate.mmd | other | 118 B |
| scripts/fixtures/smoke-invalid.mmd | other | 117 B |
| scripts/fixtures/smoke-manifest.json | config | 343 B |
| scripts/fixtures/smoke-results.jsonl | other | 755 B |
| scripts/run_openrouter_matrix.pyexecutable | script | 16.2 KB |
| scripts/score_candidates.pyexecutable | script | 10.6 KB |
| scripts/summarize_benchmark.pyexecutable | script | 23.4 KB |
| scripts/validate_candidates.tsexecutable | script | 2.9 KB |
references/methodology.md
View on GitHub# L1/L2 Methodology and Commands
## Run manifest
Save the following to `/tmp/ai-repair-benchmark/<run-id>/manifest.json`:
```json
{
"dslPath": "/absolute/path/input.dsl",
"errorMessage": "real parser error",
"diagramType": "mermaid",
"languageKey": "LANG_MERMAID",
"subTypeKey": "GENERAL",
"command": "Please fix the reported error and change nothing else.",
"models": ["openai/gpt-5.6-luna", "anthropic/claude-sonnet-5"],
"requiredAnchors": [],
"temperature": 0.2,
"maxTokens": 10000,
"seed": 20260919
}
```
Do not copy the DSL contents into the manifest. Record both repository HEADs, working-tree status, prompt hash, model parameters, trial count, and seed. Do not merge results produced with different prompt or code versions. For model-only comparisons, `temperature`, `maxTokens`, and `reasoningDisabled` may be omitted. For an unset `AI_DISABLE_REASONING`, the runner follows the default in Diagramly `ai-config.ts`: reasoning is enabled only when explicitly set to `false`; otherwise `reasoningDisabled: true`. **When validating the product AI Repair or AI Chat path, the derived L1/L2 manifest must explicitly include the same `reasoningDisabled` boolean used by L3/L4.** AI Repair reads `resolveRepairAIOptions` (currently defaults to `true`); AI Chat reads explicit overrides in the versioned worker and the provider defaults in `ai-config.ts` (currently defaults to `true`). Check L1/L2 model-call records, L3 requests and job output, and L4 UI requests and target-environment configuration. Results with mismatched settings cannot be compared across layers or counted as a full-path PASS. Keep effective parameters fixed within each comparison group.
## Generate the L2 prompt
Run from the Diagramly checkout, loading the same `.env` as the local service:
```bash
pnpm exec dotenv -c -- pnpm exec tsx --tsconfig scripts/tsconfig.json \
../conf-app/.claude/skills/ai-repair-benchmark/scripts/build_diagramly_prompt.ts \
--manifest /tmp/ai-repair-benchmark/<run-id>/manifest.json \
--output /tmp/ai-repair-benchmark/<run-id>/diagramly-messages.json \
--diagramly-path "$PWD"
```
The output includes the actual messages, prompt hash, Diagramly commit, hashes of prompt/preprocessing hook source files, and cache-control settings. If generation fails, do not fall back to a hand-written approximation of the prompt.
## One model call for one layer
First use the same command with `--dry-run` to check the plan. Run the following from the Diagramly checkout. The example calls Luna once; change `--model` and use a separate output directory to test other models. Score each call before making the next model request. A real call sends the full DSL to OpenRouter and does not print the key.
```bash
pnpm exec dotenv -c -- uv run \
../conf-app/.claude/skills/ai-repair-benchmark/scripts/run_openrouter_matrix.py \
--manifest /tmp/ai-repair-benchmark/<run-id>/manifest.json \
--mode raw --model openai/gpt-5.6-luna --trials 1 \
--output-dir /tmp/ai-repair-benchmark/<run-id>/l1-luna
pnpm exec dotenv -c -- uv run \
../conf-app/.claude/skills/ai-repair-benchmark/scripts/run_openrouter_matrix.py \
--manifest /tmp/ai-repair-benchmark/<run-id>/manifest.json \
--mode diagramly \
--messages /tmp/ai-repair-benchmark/<run-id>/diagramly-messages.json \
--model openai/gpt-5.6-luna --trials 1 \
--output-dir /tmp/ai-repair-benchmark/<run-id>/l2-luna
```
Immediately after each command, run the scoring command below, pointing `--results` and `--output` to that call's directory:
```bash
uv run .claude/skills/ai-repair-benchmark/scripts/score_candidates.py \
--manifest /tmp/ai-repair-benchmark/<run-id>/manifest.json \
--results /tmp/ai-repair-benchmark/<run-id>/l1-luna/results.jsonl \
--diagramly-path ../diagramly.ai \
--output /tmp/ai-repair-benchmark/<run-id>/l1-luna/scored.jsonl
```
`repairStatus: PASS` from the scoring script means only that its deterministic syntax check passed. Before aggregation, check actual rendering for the applicable language and inspect `missingRequiredAnchors`. The deterministic Node parser result for Mermaid does not replace Mermaid parse/render in a browser. Write the observed render result to `renderValid` in the trial record and preserve evidence in `evidencePaths`. If this field is missing, the aggregator marks the record `UNVERIFIED/SKIPPED`; do not call it a complete repair PASS.
Allow only one model request per trial. Record network failures separately; do not silently retry in the client. If only response extraction logic needs repair, use `--reparse-only` to rebuild candidates offline from saved response JSON without another paid call. After an interruption, use `--resume`, retaining interruption records and not overwriting existing results. Record the actual provider, generation id, prompt/completion/reasoning/cache tokens, and cost provenance; report cache hits separately in raw latency analysis.
## Aggregate
Pass the scored JSONL files from the L1/L2 runs; do not pass L3/L4 files:
```bash
uv run .claude/skills/ai-repair-benchmark/scripts/summarize_benchmark.py \
--manifest /tmp/ai-repair-benchmark/<run-id>/manifest.json \
--input /tmp/ai-repair-benchmark/<run-id>/l1-luna/scored.jsonl \
--input /tmp/ai-repair-benchmark/<run-id>/l2-luna/scored.jsonl \
--html /tmp/ai-repair-benchmark/<run-id>/report.html
```
Add each other model's scored file as another `--input`. Local/private reports show the full input DSL, diagram type, real error, line count, character count, and SHA-256 by default. Add `--redact-dsl` only when the user explicitly requests a public/redacted report. State n for each group; p95 does not indicate stable performance when n is small.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Unverified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |