ZenUml/conf-app / .claude/skills/ai-repair-regression
ai-repair-regression
VerifiedRun the current Prompt against saved original AI Repair error DSL in the private corpus. Simple regression runs L2 only; full regression validates L2→L3→L4. Use for “AI Repair simple/full regression test” or replaying historical cases after a repair batch; it does not collect new production failure Jobs or compare models.
AI Repair Historical Corpus Regression
This skill replays only saved original failure DSL and checks whether the current code, Prompt, and default model reproduce old problems. When the user asks specifically for an “AI Repair simple regression test,” use this skill directly; no production time range is needed, do not query production Jobs/Mixpanel or import new cases, and do not require historical cases to have previously passed L4. Use ai-repair-improvement to find and fix newly observed production failures; use ai-repair-case-validation for single-case diagnosis; use ai-repair-benchmark for model comparisons.
Determine scope and authorization
- Locate the conf-app and diagramly.ai checkouts and verify their remotes, project instructions, HEADs, and workspace states. Discover saved cases from the immediate subdirectories of
private/customer-data/ai-repair-regression/cases/; read each case's manifest, source metadata, original DSL, actual errors,expected.jsonwhen present, and any historical results. Do not require a corpus-level index:corpus.jsonmay be absent. Verify each input SHA-256 and prior PASS/FAIL/unverified status. If an original file is missing, report a blocker; do not reconstruct customer DSL from public files or reports. Respect a saved case'sexcludeFromFutureRegression: truemarker incorpus.jsonor its case metadata: omit it from model calls and validation even when the request says "all cases," unless the user explicitly re-includes that case. Report excluded cases and the eligible count separately; preserve their original files and historical results. If no case scope is specified, use all eligible saved cases and first tell the user the count. If a maximum count is specified, sample to cover earliest failures, distinct DSL types, and recent regressions, and list cases not covered. - “Simple regression” means L2 only; “full regression” means L2 → L3 → L4. Ask for a mode only when the user says “regression” without specifying one. Default to one model call per case. Model calls have no preset count, token, or cost cap; do not ask for one. Before a call, check whether authorization covers data transmission for the selected cases, actual AI service, route, and model. Explain where the complete original DSL, actual errors, and current Prompt will be sent through local Diagramly. If the current session does not have explicit authorization covering this run, ask first. An authorization recorded in an old run file does not authorize a new call. Record actual usage and cost source after each call; do not retry a failed regression case automatically because that would mask the first-run result.
- Full regression's L4 puts customer DSL into a test Confluence macro, which is a separate data destination. Separately confirm the specific site and whether a draft will remain before execution. AI service authorization does not cover this step. Without authorization, mark L4
BLOCKED (no_ui_data_authorization).
Execution and evidence
Create a record for this run at private private/local-data/ai-repair-regression/runs/<run-id>/run.json, including selected cases, mode, authorization scope, model-call usage ledger, model and destination, versions of both repositories, per-case status, and evidence paths. Set production time window and aggregate counts to not_applicable. Do not overwrite an old run.json at the corpus root or historical results. Keep original DSL, candidates, responses, and screenshots only in the local private directory; do not put them in the public repository, PRs, task titles, or ordinary messages.
Read the current DEFAULT_AI_REPAIR_MODEL from src/utils/aiModelConfig.ts. Generate the actual Prompt from the current Diagramly checkout using the L2 method. For each case, record Prompt hash, effective parameters, and raw input hash. Do not copy the Prompt by hand, rewrite the error DSL, or pass reference-fixed.* to the model. reasoningDisabled applies to AI Repair L1–L4 (verify the layers being run) and AI Chat L1–L4 as well. First read the current Diagramly resolveRepairAIOptions and explicitly record its effective product value (currently defaulting to true) in this run's derived manifest. L1/L2 model-call records, L3 request and Job output, and L4 UI request and target environment must all agree with that value. If the L4 request field is not directly visible, establish the effective value using the backend default and environment configuration. Without evidence, do not mark the entire chain PASS. Mark results with mismatched parameters as invalid configuration; do not advance to the next layer. Log each call in the shared usage ledger. Immediately save the raw response, candidate, and token/cost source after the call, then validate that case. Do not add calls automatically after failure.
L2 PASS requires all of the following: a non-empty candidate that is not a return of the original text, empty missingRequiredAnchors, syntax checks passing for the applicable language, and evidence of actual local rendering. Follow the L2 browser rendering procedure: Mermaid uses the version installed in current Diagramly; PlantUML uses an offline local validator and renderer. If a validator is unavailable, mark UNVERIFIED; a successful model response or preliminary score_candidates.py score is not a substitute for rendering. Preserve each case's candidate, check results, visible preview, and failure evidence. Without a visible preview, do not mark L2 PASS.
For expected.json cases marked frontend_false_positive, validate that the untouched original DSL renders and run the named frontend guard test. Record VALID_ORIGINAL separately from L2 repair passes and make no AI Repair model call. In a full regression, L4 verifies that the editor shows the original diagram without a stale syntax error or AI Repair trigger; it does not require Apply Code. Do not count this kind as a model-repair PASS or FAIL.
For simple regression, stop after L2 and mark each case's L3/L4 SKIPPED (simple_regression). For full regression, follow the per-layer gates in ai-repair-case-validation: run L3 only after L2 PASS with the same model, and L4 only after L3 PASS. L4 requires actual Forge UI error state, diff, post-Apply preview, and network evidence, and must reuse the currently logged-in browser under project rules. If any layer fails or is blocked, stop that case's later layers, preserve the reason and evidence, and do not bypass the gate with a historical PASS.
Compare each case's historical state with this run's result. If a regression or failure appears, preserve the raw output and report the earliest failing layer. The regression task itself does not modify the Prompt/code or add model calls. If the user requests a fix, switch to the relevant case-validation or continuous-improvement flow.
Delivery
Report covered and uncovered cases, each case's current PASS/FAIL/UNVERIFIED/BLOCKED status, historical comparison, model and Prompt/input versions, call count, cost, and private evidence paths. A passing simple regression describes only the current run's L2 results; a historical L4 preview is not evidence for this run's L4, and one successful run does not prove stability across repeated samples. Report only what was observed; do not infer an overall success rate from a small sample. Unless explicitly requested separately, do not commit, push, deploy, publish pages, or schedule recurring runs.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |