Skip to content

ZenUml/conf-app / .claude/skills/ai-chat-case-validation

ai-chat-case-validation

Verified

Validate one ordinary AI Chat diagram modification instruction by checking model output, a local versioned job, and the Confluence editor UI layer by layer; use for failure diagnosis and retesting after fixes. For syntax-error-driven repair, use ai-repair-case-validation.

AI Chat Case Validation

This skill handles ordinary modifications: the user provides a non-empty command and no errorMessage. AI Chat syntax-error repair is out of scope; the legacy AI Repair /fix-diagram entry point has its own ai-repair-case-validation workflow. Do not mix the two paths. A case fixes the original DSL, instruction, and acceptance criteria; do not manufacture a PASS by rewriting the input or relaxing requirements. Read the case format before running.

Layer gates

  1. L2 Model and Prompt: Generate the product's actual Prompt from the current Diagramly checkout and run one modification with the product-default AI Chat model. The candidate must be returned without a call error and must pass parsing and actual rendering for the applicable language; check each requested change and every item that must be preserved. Successful parsing is only a technical pass, not proof that the instruction was fulfilled. Do not proceed to L3 if L2 is FAIL or UNVERIFIED.
  2. L3 Local versioned job: Use the local Diagramly API through ensure-diagram → modify-version-async → job-status. A jobId, terminal COMPLETED status, and no job/API Error are required. For an expected modification, diagramCode and versionId must also be non-empty. Repeat L2's rendering and semantic checks. Do not proceed to L4 if L3 fails or is blocked.
  3. L4 Live Forge UI: Submit an ordinary AI Chat request in the Confluence editor using the same original DSL and instruction. Confirm the request returns, the job completes, the editor receives modified code, and the preview is visible; also confirm this flow has no relevant page errors, parse errors, network failures, or console Errors. Save a UI snapshot/screenshot and key network status. Unit tests and L3 results cannot replace L4 UI evidence.

Record PASS, FAIL, UNVERIFIED, or BLOCKED for each layer; mark later layers not reached as SKIPPED (blocked_by_L2/L3). An expected no_change may pass only when the case explicitly says the original diagram already satisfies the instruction and this has been manually verified; do not automatically treat every no-op as a failure or success. One successful run does not demonstrate model stability.

Preparation and calls

reasoningDisabled is an effective parameter for L1–L4 in both AI Chat and AI Repair; this skill starts at L2. Ordinary AI Chat reads the provider default from Diagramly ai-config.ts (currently, reasoning is enabled only when AI_DISABLE_REASONING=false); Fix syntax also reads resolveRepairAIOptions. Explicitly write the boolean value into this run's derived manifest, then verify the L2 model-call record, L3 request and job output, and L4 UI request and target backend configuration. If the UI omits the field, determine its value from the backend default; do not equate a missing field with false. Results with different effective values cannot be combined into one PASS chain.

  • Check the current code, Prompt, model configuration, and worktree state in conf-app and diagramly.ai. Read the default model from DEFAULT_AI_CHAT_MODEL in src/utils/aiModelConfig.ts; do not reuse the AI Repair model key. Record the DSL and instruction SHA-256 hashes, code/Prompt versions, effective parameters, and target environment. Also record any existing local ai_chat_model override if it affects the actual UI request.
  • Store the original DSL, instruction, model response, candidate, and screenshots only in /tmp/ai-chat-case-validation/<run-id>/ or private/local-data/ai-chat-regression/. Do not put customer content in the public repository, PR, task title, or ordinary messages. Before sending data to an external model, verify the user's authorization for this case, service, model, and call budget. Customer DSL placed in a test Confluence macro requires separate confirmation of the site and draft handling.
  • For L2, follow Product Prompt procedure to reuse the AI Repair benchmark builder and single-model runner. The manifest must provide a real ordinary-modification command, must omit errorMessage, and must use only --mode diagramly. The existing builder's final user turn says “repair instructions,” but it matches the current Diagramly buildModifyMessages; verify that source implementation before each run, and update the build procedure if it has changed. Do not hand-write an approximate Prompt. A PASS from score_candidates.py is only a preliminary syntax score and cannot replace rendering or semantic acceptance.
  • For L3, follow Local job procedure and use scripts/run_versioned_job.py. It creates a test Diagramly diagram and version in the local database; do not use a production service URL as the local test address. For L4, follow the project browser rules: first attach to the currently signed-in Chrome; if that is unavailable, use the Codex built-in browser at the standard desktop viewport. Operate the browser serially for each case. Do not substitute the existing AI Repair /diagramly/fix-diagram path for /diagramly/chat-modify.

Failure loop and deliverable

When a failure occurs, first save the raw response, error, candidate, and evidence from the earliest failing layer. Then fix the controllable Prompt, code, parameters, or local environment and retest from the earliest affected layer using the same input and instruction. Check remaining budget before every model call. If semantic acceptance criteria are missing, mark UNVERIFIED (missing_expectation) and ask the user or case evidence to supply them; do not declare success just because a response exists and has no Error.

Report each layer's status, instruction fulfillment, error-free and rendering evidence, versions, call count and cost source, and the actual UI preview path. The current product automatically updates the editor after generating a result; this does not mean the user accepted it or that it was saved to Confluence. Perform a separate persistence check only if the user requests it, and distinguish “editor updated” from “saved to Confluence.” This skill does not authorize automatic commits, pushes, deployments, or page publishing.

brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 321kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 197kUpdated Claude CodeCodexVerifiedInstall