ZenUml/conf-app / .claude/skills/ai-repair-benchmark
ai-repair-benchmark
VerifiedCompare how OpenRouter models repair the same damaged DSL; run only L1 raw-model and L2 Diagramly Prompt trials, measuring repair quality, latency, tokens, and actual cost. Use ai-repair-case-validation to investigate and regress specific cases.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 4.6 KB |
| agents/openai.yaml | config | 275 B |
| references/methodology.md | markdown | 5.6 KB |
| scripts/build_diagramly_prompt.tsexecutable | script | 4.3 KB |
| scripts/fixtures/smoke-candidate.mmd | other | 118 B |
| scripts/fixtures/smoke-invalid.mmd | other | 117 B |
| scripts/fixtures/smoke-manifest.json | config | 343 B |
| scripts/fixtures/smoke-results.jsonl | other | 755 B |
| scripts/run_openrouter_matrix.pyexecutable | script | 16.2 KB |
| scripts/score_candidates.pyexecutable | script | 10.6 KB |
| scripts/summarize_benchmark.pyexecutable | script | 23.4 KB |
| scripts/validate_candidates.tsexecutable | script | 2.9 KB |
scripts/validate_candidates.ts
View on GitHub#!/usr/bin/env -S pnpm exec tsx
import { readFileSync, writeFileSync } from "node:fs";
import { join, resolve } from "node:path";
import { pathToFileURL } from "node:url";
type Item = { id: string; candidatePath: string };
function arg(name: string): string {
const index = process.argv.indexOf(name);
if (index < 0 || !process.argv[index + 1]) {
throw new Error(`Missing ${name}`);
}
return process.argv[index + 1];
}
async function main() {
const manifestPath = resolve(arg("--manifest"));
const outputPath = resolve(arg("--output"));
const diagramlyPath = resolve(arg("--diagramly-path"));
const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as {
languageKey: string;
items: Item[];
};
const parserPath = join(
diagramlyPath,
"apps/web/modules/saas/diagram/services/deterministic-syntax-parser.ts",
);
// The app's parser imports workspace aliases that tsx may resolve through a
// CommonJS boundary when this external helper runs. Mermaid's parser is
// headless and the same version is installed in Diagramly's web package, so
// use it directly for Mermaid syntax checks. This is still a syntax check;
// browser rendering remains a separate acceptance gate.
let mermaid: (typeof import("mermaid"))["default"] | null = null;
if (manifest.languageKey === "LANG_MERMAID") {
const sanitizerPath = join(
diagramlyPath,
"apps/web/lib/ai/mermaid-server-sanitizer.ts",
);
const { prepareMermaidServerSanitizer } = await import(
pathToFileURL(sanitizerPath).href
);
await prepareMermaidServerSanitizer();
// Use the package entry (mermaid.core.mjs), not the self-contained
// mermaid.esm.mjs bundle: the bundle embeds another DOMPurify instance
// that the server sanitizer cannot prepare.
mermaid = (await import(pathToFileURL(join(
diagramlyPath,
"apps/web/node_modules/mermaid/dist/mermaid.core.mjs",
)).href)).default;
}
const deterministicSyntaxParser = mermaid
? null
: (await import(pathToFileURL(parserPath).href) as {
deterministicSyntaxParser: (
code: string,
languageKey: string,
) => Promise<string | null>;
}).deterministicSyntaxParser;
const results: Array<Record<string, unknown>> = [];
for (const item of manifest.items) {
const code = readFileSync(resolve(item.candidatePath), "utf8");
if (manifest.languageKey === "LANG_PLANTUML") {
results.push({ id: item.id, skipped: true, error: null });
continue;
}
let error: string | null;
if (mermaid) {
try {
await mermaid.parse(code);
error = null;
} catch (cause) {
error = cause instanceof Error ? cause.message : String(cause);
}
} else {
error = await deterministicSyntaxParser!(code, manifest.languageKey);
}
results.push({ id: item.id, skipped: false, error });
}
writeFileSync(outputPath, `${JSON.stringify(results, null, 2)}\n`);
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exitCode = 1;
});
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |