Skip to content

gracefullight/pkgs / .agents/skills/oma-qa

oma-qa

Verified

Review changes for correctness, security, accessibility, and performance. Use for scoped quality reviews or verification plans; bug fixes use oma-debug.

resources/execution-protocol.md

View on GitHub
# QA Agent - Execution Protocol

## Preparation
Use the task's scope, existing project conventions, and acceptance criteria. Follow `../../_shared/core/execution-policy.md` when it has not already been supplied. Read only references needed by the selected operation; consult lessons or recovery guides for an observed issue. Expand planning depth only when the change requires it.

## Step 1: Scope
- Identify what to review: new feature, full audit, or specific concern
- List all files/modules to inspect
- Determine review depth: quick check vs. comprehensive audit
- Follow `../../_shared/core/code-intelligence.md` to discover configured code
  navigation/search tools. If unavailable or timed out, use native search only
  for paths outside this project or ignored paths, record that limit, and continue the review.

## Step 2: Audit
Review in this priority order:
1. **Security** (CRITICAL): OWASP Top 10, auth, injection, data protection
2. **Performance**: API latency, N+1 queries, bundle size, Core Web Vitals
3. **Accessibility**: WCAG 2.2 AA, keyboard nav, screen reader, contrast
4. **Code Quality**: task-defined coverage expectations, complexity, architecture adherence

When applicable, map findings and gaps to:
- **ISO/IEC 25010** quality characteristics
- **ISO/IEC 29119** test planning, design, traceability, and exit criteria

Use `resources/checklist.md` (renamed qa-checklist) as the comprehensive review guide.

## Step 2.5: Runtime Verification

Static code review misses entire categories of bugs: display-only features,
stubbed functionality, broken user flows, and edge cases that only surface
at runtime. This step requires interacting with the running application.

### When to Execute
- Web app with UI: ALWAYS for Medium/Complex tasks
- API-only: ALWAYS (curl/httpie verification)
- Simple tasks (single file, no UI): SKIP

### Execution by App Type

#### Web Applications

Browser verification uses the installed MCPs selected in `mcp.devtools_browsers`: Aside (`aside`, default), Chrome DevTools MCP (`chrome`), and Firefox DevTools MCP (`firefox`). Multiple selections are supported; use `oma update mcp` to change them. Discover the selected server’s actual tools before use; tool names and capabilities differ between servers. An empty selection disables browser MCP verification; report any unverified UI checks.

Run the following checks with a selected, available server. Prefer a separate test context when supported. The tool calls below are Chrome DevTools MCP examples only; for Aside and Firefox DevTools MCP, discover and use their supported equivalents. Record missing capabilities instead of silently changing servers.

1. Start the application (`bun run dev`, `uv run manage.py runserver`, etc.)
2. Open the app in an **isolated browser context** to avoid contaminating the user's session:
   ```
   new_page(url: "http://localhost:PORT", isolatedContext: "qa-test")
   ```
   - Pages in the same `isolatedContext` share cookies/storage
   - Pages in different contexts are fully isolated
   - When using Chrome DevTools MCP, use `isolatedContext: "qa-test"` for QA verification
3. Navigate and inspect:
   ```
   navigate_page(url)            → navigate within the isolated context (SPA routes, sub-pages)
   take_snapshot()               → a11y tree snapshot with uid-tagged elements (prefer over screenshots)
   ```
4. Interact with modified features:
   ```
   click(uid)                    → click buttons/links
   fill(uid, value)              → fill input/textarea/select fields
   fill_form(elements)           → fill multiple fields at once
   type_text(text, submitKey)    → type into focused input
   press_key(key)                → keyboard shortcuts, Enter, Tab
   hover(uid)                    → hover for tooltips, dropdown menus, state changes
   drag(from_uid, to_uid)        → drag-and-drop
   ```
5. Verify functional correctness:
   - `list_console_messages(types: ["error", "warn"])` → detect JS errors
   - `list_network_requests()` → verify API calls fired and returned expected status
   - `get_network_request(reqid)` → inspect request/response bodies
   - `evaluate_script(function)` → check DOM state, global variables, computed styles
   - `wait_for(text)` → confirm expected content appears after interaction
6. Visual and accessibility verification:
   - `take_screenshot()` → visual output matches acceptance criteria
   - `lighthouse_audit(mode: "snapshot")` → accessibility, SEO, best practices scores
   - `emulate(viewport: "375x812x3,mobile,touch")` → responsive/mobile verification
7. Performance profiling (when performance criteria exist):
   - `performance_start_trace(reload: true)` → capture Core Web Vitals
   - `performance_stop_trace()` → analyze LCP, INP, CLS
   - `take_heapsnapshot(filePath)` → detect memory leaks
8. Cleanup:
   ```
   close_page(pageId)  → close isolated test pages after verification
   ```
9. **Fallback** (no selected browser MCP available):
   - Use curl/httpie to hit rendered endpoints
   - Verify HTTP status codes and response bodies; record that HTTP checks do not verify browser interactions
   - Check redirects, auth flows, and error pages

#### API Endpoints
1. Start the server
2. Execute acceptance criteria as actual HTTP requests:
   ```bash
   # Example: verify auth flow end-to-end
   curl -s -X POST localhost:8000/api/auth/register \
     -H "Content-Type: application/json" \
     -d '{"email":"test@test.com","password":"secure123"}'

   # Verify rate limiting actually triggers
   for i in $(seq 1 20); do
     curl -s -o /dev/null -w "%{http_code}" localhost:8000/api/auth/login
   done
   ```
3. Verify database state after operations (query directly or via API)

#### Mobile Applications
1. If emulator/simulator available: launch and interact
2. If not: verify via API layer + widget test execution

### Recording Results

Append to the QA report under a new section:

```markdown
## Runtime Verification Results

| Feature | Method | Expected | Actual | Status |
|---------|--------|----------|--------|--------|
| User registration | curl POST /api/auth/register | 201 + user created | 201 + user in DB | PASS |
| Rate limiting | 20x rapid POST /api/auth/login | 429 after threshold | 429 after 10 req | PASS |
| Empty state UI | navigate_page + take_snapshot | Empty state message | Blank white page | FAIL |
| Button handler | click(uid) + list_network_requests | POST /api/save fired | No request fired | FAIL |
| Console errors | list_console_messages(["error"]) | 0 errors | TypeError at app.js:42 | FAIL |
```

### Stubbed Feature Detection

Specifically check for these patterns that static review cannot catch:
- Buttons/forms that render but have no backend handler → `click(uid)` + `list_network_requests()` to verify
- Features that display placeholder data instead of real data → `evaluate_script()` to check data binding
- Interactive elements that don't respond to user input → `click(uid)` + `take_snapshot()` to compare before/after
- Audio/video/file upload controls with no actual processing → `upload_file(uid, filePath)` + verify response

---

## Evaluator Posture: Evidence-based

Apply this posture when making verdict decisions in Step 3 and Step 4:
- Base defect findings on reproducible evidence; track unconfirmed hypotheses separately.
- "Probably works" or "should be fine" is NOT valid evidence. Run it, show output, prove it.
- If a required feature cannot be verified at runtime, record the missing check, reason, and next action. Mark review completeness `partial`, or `blocked` when no independent verification can proceed, following the shared execution policy and host rules.
- A skipped check, missing tool, or interrupted command is not evidence that the product works or that it contains a defect. A failed verification command must not count as a passing check; diagnose whether it demonstrates a product defect or an environment/configuration problem.
- Never downgrade a real bug to "non-critical" just to pass a gate.
- Use WARNING when all remaining issues are MEDIUM or lower and none block deployment.
- When evidence is insufficient, report the verification gap and continue available checks. Do not invent a defect or issue PASS to force a binary verdict.

## Step 3: Report
Generate structured report with:
- Report both quality verdict and verification completeness:
  - PASS: required checks completed; no CRITICAL, HIGH, or MEDIUM issues
  - WARNING: required checks completed; no CRITICAL or HIGH issues, but MEDIUM issues exist
  - FAIL: a reproducible CRITICAL or HIGH issue exists, even if other checks remain unavailable
  - NOT_ASSESSED: required evidence is missing and no confirmed defect warrants FAIL; list any confirmed lower-severity findings separately
  - Completeness: `completed` when required verification is finished, `partial` when evidence remains missing, or `blocked` when independent work cannot proceed under the shared execution policy and host rules
- Verification gaps: missing check, cause, completed alternative checks, and concrete next action; keep these separate from defect findings
- Findings grouped by severity (CRITICAL > HIGH > MEDIUM > LOW)
- Each finding: file:line, description, remediation code
- Performance metrics vs. targets
- Standards suggestions when relevant:
  - quality characteristics under-covered
  - missing test design / traceability / exit criteria

## Step 4: Verify
- Use applicable sections of `../../_shared/core/common-checklist.md` when the review crosses domains

## On Error
See `resources/error-playbook.md` for recovery steps.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall