Skip to content

ZenUml/conf-app / .claude/skills/ai-repair-case-validation

ai-repair-case-validation

Verified

Investigate specific AI Repair failures, verify repairs, and regress historical cases. By default, validate the default model strictly through L2 → L3 → L4; for a stronger-model retry, validate only L2 unless continuation through L3/L4 is explicitly requested. Use ai-repair-benchmark for model comparisons.

AI Repair Case Validation

For a specific failure, run L2 → L3 → L4 on the product's default repair model by default. Use this to investigate a failure, verify a repair, or replay a historical failure. A stronger-model retry is checked at L2 only by default; if the user explicitly asks to continue that model through L3 or L4, validate each layer for that same model. For a multi-case regression, run each DSL independently and record its gate results. This skill does not run L1 or compare models; use ai-repair-benchmark for model comparisons. Use pvt-ai-repair for general smoke checks after release.

Strict gates

  1. Run a model's L3 only after that model under test passes L2. If L2 fails or is blocked by the environment, stop that model's chain, diagnose and improve L2, then rerun it. A stronger model's L2 PASS cannot substitute for the default model's L2 PASS. Run the stronger model's L3 only when explicitly requested, and only after that model passes its own L2.
  2. Run a model's L4 only after that model under test passes L3. If L3 fails or is blocked by the environment, stop that model's chain, diagnose and improve L3, then rerun it. Do not continue to that model's L4 even if the only L3 issue is that the local database is not running.
  3. Returning content identical to the original DSL (ignoring leading/trailing whitespace and newline format) is a no-op and must be recorded as repair FAIL; even if the API/job reports success or the parser is unavailable, do not mark it PASS or SKIPPED. Every other PASS requires evidence that the candidate is non-empty, parses/renders successfully for the applicable language, and satisfies required anchors. A layer is not PASS if required validation cannot be completed.
  4. Mark layers not reached as SKIPPED, recording blocked_by_L2 or blocked_by_L3 and the specific reason. Do not label an untested layer FAIL or infer its production behavior.

Apply these gates separately to each model under test. Read and save any existing evidence of production failures before starting. For repair verification and historical case regressions, start again at L2 with the product default model and proceed only after each layer passes. When another model is explicitly specified, start that model at L2 as well. Record code/prompt versions and configuration each time; keep results from different versions separate.

If the same approach fails at a layer multiple times, fix that layer first; do not let a later one-off success hide earlier failures. Record every trial and confirm the current repair approach meets that layer's gate before proceeding.

Repair loop after a failure

After a FAIL, do not deliver only a failure report. Save the original input, response, candidate, logs, and decision reason; identify the earliest failing layer; fix controllable prompts, parameters, code, or local environment; then rerun that layer with the same original DSL, error, and required anchors. An unchanged response, empty candidate, invalid syntax, missing anchors, job failure, or failed UI application must enter this loop. Do not hide a failure by rewriting the case input, weakening anchors, or switching models. Restore a blocked environment and rerun; an environment block by itself is not a model repair failure.

Record each fix, code/prompt version, and retest result separately. If a change affects an upstream layer's input or behavior, revalidate from the earliest affected layer. If it only fixes infrastructure at the current layer, retain upstream evidence that remains applicable. Keep diagnosing and improving after repeated failures; do not let a one-off success override earlier failures. Proceed only after the current layer meets its gate. If a fix is impossible or required authorization/environment is unavailable, report attempted fixes and the blocker, and leave later layers SKIPPED. This loop does not authorize automatic commits, deployment, or publishing.

Preparation

  • For an initial investigation, collect the original production DSL, actual parser error, diagram type, model used by the user, time, UI behavior, fix-diagram / job-status responses, and relevant logs. For repair verification or a historical regression, read the saved original input, error, expected required anchors, and previous failure evidence. Separate observed facts, unknowns, and hypotheses to validate; do not treat local results as production facts.
  • Before each run, read src/utils/aiModelConfig.ts: use DEFAULT_AI_REPAIR_MODEL for the default path and AI_REPAIR_RETRY_MODEL for a stronger-model retry. If the case record or local ai_repair_model override indicates another model was actually used, record the source. By default, run a separate L2 check for that model; continue through L3/L4 only when explicitly requested. Do not treat example strings in this skill or its scripts as product defaults, and never pass one model's output to another model.
  • Check HEAD, working-tree status, prompt hash, model parameters, and target environment for conf-app and Diagramly. Preserve existing changes; this skill does not automatically create branches, commit, deploy, or publish pages.
  • Save this run's manifest, candidates, and evidence under /tmp/ai-repair-case-validation/<run-id>/. When reusing commands from the benchmark docs, replace /tmp/ai-repair-benchmark/<run-id>/ in examples with this directory.
  • Before sending the full DSL to OpenRouter, obtain the user's explicit authorization if it is from production or may contain customer information. Store customer data and evidence only in /tmp or the private/ submodule, never in the public repository. Do not save tokens or authorization request headers.

L2: Prompt and model

Reuse the benchmark L2 methodology and build_diagramly_prompt.ts, run_openrouter_matrix.py, and score_candidates.py. Create a single-model manifest with DEFAULT_AI_REPAIR_MODEL, and pass --model <default-model> and --trials 1. Before the call, read the current Diagramly resolveRepairAIOptions and explicitly write the product-effective reasoning setting into this run's derived manifest (currently reasoningDisabled: true); do not rely on the benchmark runner's .env fallback. Check the result fields after the call; if they do not match, mark the configuration invalid and do not allow L3. Create a separate L2 record with the same original DSL and AI_REPAIR_RETRY_MODEL only when the case involves a stronger-model retry or that retry needs diagnosis. By default, this result does not start L3/L4; it becomes the L3 gate for that stronger model only when continuation is explicitly requested. Generate the actual messages from the current Diagramly checkout; do not copy the prompt by hand. Validate each candidate immediately after its call, and record the raw response, prompt hash, model parameters, parse/render results, and required anchors. A PASS from score_candidates.py is only an initial score: also confirm missingRequiredAnchors is empty and obtain actual rendering evidence for the applicable language. For Mermaid, follow L2 browser rendering evidence with the Mermaid version in the current checkout. Without this evidence, the model's L2 cannot pass the gate for L3. Repeat trials only when needed to measure random variation and record each trial; if the model has not met the L2 gate, stop at L2 and improve it instead of starting L3 early.

L3: Local asynchronous job

Read the L3 operating details only after the model under test passes L2. Use the product default model unless another model is explicitly specified through --model or a single-model manifest. Prepare local Diagramly and PostgreSQL using joint-debug. Use run_job_matrix.py to create and poll one local repair job, then validate its result with the benchmark's score_candidates.py. Compare against the same model's L2 record for original DSL, model, error, and key settings. Explicitly check that reasoningDisabled in the job output matches L2; explain any mismatch first and do not count results with different settings as the same path passing. Record jobId, status, poll count, candidate, and failure reason. An environment check failure is not a PASS; that model's L4 remains SKIPPED (blocked_by_L3).

L4: Real Forge UI

Read the L4 operating details only after the model under test passes L3, and follow the UI evidence rules in joint-debug and pvt-ai-repair. Validate the product default model unless another model is explicitly specified; verify that the request actually uses that model. Reuse the user's currently logged-in browser tab; do not create a browser session for a real account. If the current browser cannot be operated, follow project instructions to try the Codex built-in browser at the standard desktop viewport. Restore the same damaged DSL and check that the effective reasoningDisabled value in the UI request and target backend matches L2/L3. Then verify the error banner, job creation and polling, diff, and that Apply Code clears the error and produces a valid preview. If the request omits the field, use the backend default; do not treat an omitted field as false. Do not mark the full path PASS unless the effective value can be proven. Do not mark PASS without actual UI screenshot, snapshot, or network evidence.

Conclusion and delivery

For each model under test and each layer, list the input version, PASS/FAIL/SKIPPED, evidence, earliest failing stage, fixes, and retest results. A stronger-model retry has only an L2 result by default; include its L3/L4 only when continuation was explicitly requested. For an initial investigation, describe the failure as seen by the user first, followed by runtime evidence and code path. For a historical regression, also compare the previous failure point with the current result. L2/L3 success does not prove the UI has recovered in the target environment; only actual L4 UI evidence proves that model's repair path passed in this run. Keep untested layers SKIPPED and state the gating blocker. Do not estimate overall model success rates from one success or failure.

slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall