Skip to content

dimagi-internal/ace / skills/app-ux-eval

app-ux-eval

Verified

Internal /ace:run step — Grade FLW app UX from screenshots against the journeys.

FileKindSize
SKILL.mdskill md19.7 KB
---
name: app-ux-eval
description: >
  Internal /ace:run step — Grade FLW app UX from screenshots against the journeys.
user-invocable: false
disable-model-invocation: false
---

# App UX Eval

Grades the FLW experience of the built apps. Asks: "would this be a
good experience for the user?" and pins each judgment to concrete
PDD-derived ground truth (the journey's stated goal, time budget, edge
cases) so the rubric isn't unmoored.

This skill is the **eval** half of the app deep-QA pair — it does not
drive the AVD or capture screenshots. For the capture half, see
`app-screenshot-capture` (run as part of `/ace:qa-deep`). See
`skills/_eval-template.md` for shared verdict / severity / stock-block
contracts.

## Inputs

Pipeline-only: grades a run's Phase 6 screenshots against its Phase 2 journeys, which exist only in a run.

| Source | Artifact | Used for |
|---|---|---|
| Phase 1 | `2-scenarios/pdd-to-app-journeys.md` | UX-intent ground truth (goal, happy-path narrative, edge cases, pass criteria, `pdd_time_budget_seconds`) |
| Phase 3 | `3-commcare/app-test-cases.yaml` | journey↔recipe bindings, smoke flag, forms exercised |
| Phase 6 | `6-qa-and-training/app-screenshot-capture_manifest.yaml` → each frame by `file_id` (smoke frames live in `3-commcare/previews/<app-output-slug>/`, deep frames in `6-qa-and-training/screenshots/<recipe-base>/`) | captured PNGs to grade |

## Products

- `6-qa-and-training/app-ux-eval_verdict-deep.yaml` — per-journey verdict on UX dimensions (clarity, flow_predictability, error_recovery, time_budget, journey_completion, capture_robustness)

## Process

### Step 1: Read inputs

- `2-scenarios/pdd-to-app-journeys.md` — ground truth (UX-intent:
  per-journey goal, happy-path narrative, edge cases, pass criteria,
  `pdd_time_budget_seconds`)
- `3-commcare/app-test-cases.yaml` — journey↔recipe bindings (which
  Maestro recipe corresponds to which journey, smoke flag, forms
  exercised)
- The captured screenshots from the recent execution run — look up by
  the `--run-id` argument passed in. Resolve every frame through
  `ACE/<opp>/runs/<run-id>/6-qa-and-training/app-screenshot-capture_manifest.yaml`
  (which step → which PNG, by `file_id`) — never by listing a folder: smoke
  frames sit in `3-commcare/previews/<app-output-slug>/` (output previews
  contract v1) and `/ace:qa-deep`'s deep frames in
  `6-qa-and-training/screenshots/<recipe-base>/`
- `inputs/pdd.md` — for persona context (the FLW the rubric is judging
  "good experience" against; pulled from the "Target FLW" section)
- `3-commcare/app-deploy_summary.md` — for the `learn_build_id` and
  `deliver_build_id` that go into `artifact_refs` so the Phase 9 gate
  can compare verdict freshness against the latest released CommCare
  build

### Step 2: For each journey, score 6 dimensions (1–3)

Each dimension scores 1, 2, or 3 (1 = fail, 2 = warn, 3 = pass) on the
journey's screenshot sequence. The hard-deduction column is the
single-rule that clamps the dimension to 1 (fail) regardless of how
the rest of the journey looks.

| Dimension | What to look for | Hard deduction → fail |
|---|---|---|
| `clarity` | Field labels and prompts unambiguous to the persona from PDD's "Target FLW" section. The FLW should know what's being asked without guessing. | Any field name only a developer would understand (e.g., `q3_v2_optional`, `field_3a`) appears in a screenshot |
| `flow_predictability` | Conditional branches go where the FLW expects; skip patterns don't surprise. The next screen follows from the current one. | A screen appears or disappears with no apparent cause from the user's perspective (e.g., a question shows in run 1 but not run 2 with no visible trigger) |
| `error_recovery` | Validation errors tell the FLW what's wrong and how to fix. The FLW can recover without losing prior input. | Dead-end errors with no recovery path (FLW gets stuck, or has to restart the form to retry) |
| `time_budget` | Step count + estimated input time vs. journey's `pdd_time_budget_seconds`. Recipe step count × 5s is the heuristic estimate (5s per tap/type). **Being far UNDER budget is NOT a win** (fixed 2026-05-29): a recipe whose step count × 5s is < 50% of the budget is *suspiciously thin* for its declared scope — score this dimension ≤2 and surface `[WARN] time_budget: recipe far under budget — likely under-built form, not efficiency`. Do NOT award a high score for being fast because the form is thin (the old rule only penalized "too slow," which rewarded the ITN-style skeletal build). The ideal is *near* budget. | Recipe step count × 5s exceeds 2× the budget |
| `journey_completion` | Recipe accomplishes the journey's stated goal end-to-end. Final screenshot shows confirmation / submission success / explicit completion state. | Recipe ends without confirmation / stuck screen / mid-form (the journey didn't finish) |
| `capture_robustness` | **Fitness dimension (added 2026-05-29).** Does the instrument *refuse bad data*? Graded on the **negative-path** journeys (blank required field, out-of-range value, low-accuracy GPS, "Other" selection) — does the form reject the bad input with a clear, recoverable error, or silently accept garbage? This is the screenshot-side complement to `pdd-to-deliver-app-eval § data_quality_validation`. **Coverage cap (mirrors `ocs-chatbot-eval`'s adversarial-coverage cap):** if `app-test-cases.yaml` contains ZERO negative-path journeys, this dimension is *unmeasured* — score it ≤2 (NOT pass) and surface `[WARN] capture_robustness: no negative-path journey in the test suite — the instrument's data-quality enforcement is untested`. An app whose test suite never feeds it a bad input has unproven robustness; that absence must not read as success. (Forcing function: `app-test-cases` should emit ≥1 negative-path recipe per credit-bearing form.) | Form accepts a known-bad input (blank required / out-of-range / low-GPS) with no error, OR zero negative-path journeys exist to test it |

**NOT REACHED criteria cap `journey_completion` at 2 (warn). They do
not fail the journey, and they never count as a pass (dimagi-internal/ace#2670).**
`/ace:qa-deep` Stage B step 4 hands over a NOT REACHED list. It holds every
`reachability: multi-day` criterion (N dated records on one case under a
strictly-increasing, `<= today()` date constraint, which one session cannot
produce), plus any criterion the walk did not get to for another stated
reason. Re-derive the list as well: walk each journey's
`structural_pass_criteria` and check that each one is either graded from a
screenshot or on the NOT REACHED list. An ungraded criterion missing from
that list is a `[WARN]` of its own, owner `HARNESS`. For each journey with
one or more NOT REACHED criteria (`reconcileNotReached` and
`capJourneyCompletion` in `lib/criterion-reachability.ts` are the
executable form of these rules; `unaccounted` is the HARNESS WARN list):
- List them under that journey's `per_item[].not_reached` as
  `{criterion, reason}`, quoting the reason verbatim.
- Score `journey_completion` **at most 2 (warn)**. The journey's walked
  part may well be complete, but its stated goal was not shown end to end.
  A NOT REACHED criterion is therefore **not** the "journey didn't finish"
  hard deduction. A calendar constraint is not a dead end, and failing
  the journey would report a product defect that was never observed.
  Scoring it 3 would report a pass on something nobody saw.
- Surface one `[WARN]` per criterion in `auto_surfaced`, owner
  `INSTRUMENT`, naming the journey, the criterion and the reason. For a
  `multi-day` criterion, add the remedy: re-run `/ace:qa-deep` on
  successive days against the same cases.

**Known platform rendering — not a product finding: stray spaces on an
id-mapping case-list label (ace#2692).** HQ compiles a Nova `id-mapping`
column as `replace(join(' ', ...), '\s+', ' ')`, which never trims, so a value
renders with a leading and/or trailing space (`"Step 1: Our Partnership "`)
while the blueprint label is clean. The app cannot trim it without dropping
the label's translations. Do not deduct `clarity` or any other dimension for
it, and do not raise it in `auto_surfaced` as a `PRODUCT` concern; if it is
visible in a screenshot you cite, say "known platform rendering (ace#2692)" in
that dimension's reason. The upstream fix is Nova's (emit HQ
`conditional-enum`, or trim the joined result).

For each dimension, write a one-sentence reason citing the specific
screenshot(s) that drove the score (e.g.
`"journey-deliver-step-04 shows field labelled q3_v2_optional"`).

After scoring 1–3 per dimension, convert to a 0–10 score via
`(score - 1) / 2 * 10` for the verdict YAML — see Step 4 for the
exact shape. Hard-deduction triggers set the dimension's converted
score to 0.

### Step 3: Aggregate

- **Per-journey verdict:** weighted average of dimensions. Weights
  (updated 2026-05-29 to seat the fitness dimension): `clarity` 0.15,
  `flow_predictability` 0.15, `error_recovery` 0.15, `time_budget`
  0.10, `journey_completion` 0.15, `capture_robustness` 0.30. Hard-
  deduction on any single dimension clamps the journey to **fail**
  regardless of weighted-average math —
  surface the triggering rule in the per-item `note`. Per-journey
  verdict is binary (`pass | fail`) for this skill — no warn tier.
  The verdict-schema permits `warn`, but app-ux-eval's rubric is
  pass-or-fail; gate decisions need an unambiguous green/red signal.
- **Hard-deduction effect on dimension scores:** when a dimension's
  hard-deduction triggers, that dimension's score is set to 0 in the
  YAML output AND the per-journey verdict is clamped to fail.
  `overall_score_pre_cap` retains the un-clamped weighted mean for
  calibration analysis.
- **Phase verdict:**
  - `pass` = all journeys pass
  - `fail` = any journey fails. Summary lists which journeys failed
    which dimensions, including the smoke journey vs. non-smoke
    distinction (smoke failure = blocker; non-smoke failure may
    iterate)
  - Either way, the summary **names every NOT REACHED criterion**
    (journey, criterion, reason). A `pass` with unreached criteria is a
    pass on what was observed, and it must say what was not observed.

### Step 4: Write verdict

Write
`ACE/<opp>/runs/<run-id>/6-qa-and-training/app-ux-eval_verdict-deep.yaml`
per the uniform verdict shape (see `skills/README.md § Eval verdict
shape` or `lib/verdict-schema.ts`):

```yaml
skill: app-ux-eval
target: <opp-name>
mode: deep
ran_at: <ISO timestamp with timezone>
capture_path: 6-qa-and-training/app-screenshot-capture_manifest.yaml

artifact_refs:
  learn_build_id: <from 3-commcare/app-deploy_summary.md>
  deliver_build_id: <from 3-commcare/app-deploy_summary.md>

overall_score: 7.0             # 0–10 scale; convert per-dimension 1–3 to 0–10 via (avg-1)/2*10
overall_score_pre_cap: 7.7     # raw weighted mean before any hard-deduction clamp
verdict: fail                  # pass | fail

dimensions:
  clarity:              { score: 9.0, weight: 0.15 }
  flow_predictability:  { score: 8.5, weight: 0.15 }
  error_recovery:       { score: 7.0, weight: 0.15 }
  time_budget:          { score: 7.0, weight: 0.10 }   # near-budget is ideal; far-under is a thinness WARN, not a 9.5
  journey_completion:   { score: 8.5, weight: 0.15 }
  capture_robustness:   { score: 6.0, weight: 0.30 }   # fitness: does the form refuse bad data on negative-path journeys

per_item:                # per-journey verdicts; key is the journey id (a meaningful slug) from app-test-cases.yaml
  - ref: "journey-deliver-submit"
    journey: "journey-deliver-submit"
    is_smoke: true
    score: 9.0
    verdict: pass
    note: "Final screenshot shows submission confirmation; all dimensions clear"
  - ref: "journey-deliver-alt-answer"
    journey: "journey-deliver-alt-answer"
    is_smoke: false
    score: 5.0
    verdict: fail
    note: "Hard-deduction on error_recovery: dead-end error in journey-deliver-step-07 with no recovery path (FLW must restart form). error_recovery dimension clamped to 0."
  - ref: "journey-deliver-over-cap"
    journey: "journey-deliver-over-cap"
    is_smoke: false
    score: 8.0
    verdict: pass
    not_reached:           # ace#2670 — caps journey_completion at 2 (warn); never a fail, never a pass
      - criterion: fourth_meeting_labelled_not_paid
        reason: "reachability: multi-day — needs 4 community meetings on one case; date_of_meeting must be > the previous meeting and <= today()"
    note: "Meetings 1-3 walked and labelled correctly; meeting 4 NOT REACHED (multi-day). journey_completion capped at 2."
  # ... one entry per journey in app-test-cases.yaml

auto_surfaced:
  - severity: BLOCKER | WARN | INFO
    owner: HARNESS | INSTRUMENT | PRODUCT | PROMPT
    message: <one-line concern>

not_reached:               # every per_item not_reached entry, flattened; [] when none.
  - { journey: journey-deliver-over-cap, criterion: fourth_meeting_labelled_not_paid }
                           # TOP-LEVEL on purpose: the top-level verdict schema is
                           # passthrough, GateSchema is not (zod strips unknown gate keys)

gate:
  threshold: 7.0
  disposition: approve | reject | iterate
```

Required top-level fields:

- `skill: app-ux-eval`
- `mode: deep`
- `ran_at` — ISO timestamp with timezone
- `artifact_refs: { learn_build_id, deliver_build_id }` — read from
  `3-commcare/app-release_summary.md` `apps.<app>.build_id` (the owner of
  release state, ace#1636/ace#2938) so the Phase 9 gate can compare against
  the currently released builds
- `dimensions` — per-dimension scores + reasons (6 dimensions, weights
  per the rubric above)
- `per_item` — per-journey verdicts (canonical key per
  `skills/README.md § QA vs Eval`); `ref`/`journey` is the journey's
  `journey-`-prefixed slug id from `app-test-cases.yaml` (e.g.
  `journey-learn-pass`, `journey-deliver-submit`), each entry includes a `journey` domain-specific subkey
- `overall_score`, `verdict` (`pass | fail`)
- `per_item[].not_reached` on each journey that has NOT REACHED criteria,
  and top-level `not_reached` (the flattened list, `[]` when none). An unreached
  criterion that appears in neither is invisible to the gate, which is
  the ace#2670 failure.

Also append a row to
`ACE/<opp>/eval-calibration/app-ux-eval-runs.md` (opp-level, not
run-scoped — calibration audit trails accumulate across runs) so
calibration metrics keep growing per
`skills/eval-calibration/SKILL.md`.

### Step 5: Apply the gate

- `overall_score ≥ 7.0` AND zero failing journeys = `approve`
- Any smoke-journey failure = `reject` (smoke journey is the
  primary submission flow; if that fails, the app is not launchable)
- Any non-smoke failure with overall ≥ 7.0 = `iterate` (the operator
  can decide whether to fix and re-run vs. proceed)
- NOT REACHED criteria do not change the disposition by themselves. Their
  effect is the `journey_completion` cap. The gate summary must still list
  each one, so an `approve` never hides what it did not observe.

The Phase 9 gate (`llo-launch`) reads this verdict and adds its own
freshness check (verdict's `artifact_refs` must match the latest
released build IDs — see Task 7 in the shallow/deep split plan).

## Mode behavior

- **Deep only.** There is no `--quick` mode. The shallow Phase 6
  smoke (`app-screenshot-capture` running just the `is_smoke: true`
  recipes) does NOT invoke this skill — it's structural-only.
  This skill is invoked exclusively by `/ace:qa-deep` and the Phase 9
  go-live gate.

## Failure modes

- **Screenshots missing for a journey marked in
  `app-test-cases.yaml`** → halt with a `[BLOCKER]` saying which
  recipe didn't run. Don't grade partial coverage; the missing
  screenshots are upstream signal that `/ace:qa-deep`'s capture step
  failed.
- **`2-scenarios/pdd-to-app-journeys.md` missing** → upstream Phase 1 or
  migration gap; halt with pointer to `pdd-to-app-journeys`.
- **`3-commcare/app-test-cases.yaml` missing** → upstream Phase 3 gap;
  halt with pointer to `app-test-cases`.
- **Nova builds older than the screenshots** (i.e., a CommCare build
  released after the screenshots were captured) → screenshots are
  stale; halt with `[BLOCKER]` and instruct the operator to re-run
  `/ace:qa-deep` against the current build.
- **`3-commcare/app-deploy_summary.md` missing the build IDs** → halt;
  the Phase 9 gate needs `artifact_refs` to compare freshness, and
  partial verdicts can't be timestamp-checked.

## MCP tools used

- ace-gdrive: `drive_read_file`, `drive_list_folder`,
  `drive_create_file`
- (No mobile / Nova / OCS — this is pure judging over already-captured
  artifacts)

## Change log

| Date | Change | Author |
|------|--------|--------|
| 2026-05-04 | Initial version. Deep-only LLM-as-Judge for app UX. Five dimensions (clarity, flow_predictability, error_recovery, time_budget, journey_completion) with hard-deductions. Used by /ace:qa-deep and the Phase 8 gate. Introduced as part of the shallow/deep QA split (spec: `docs/superpowers/specs/2026-05-04-shallow-deep-qa-split-design.md`). | ACE team |
| 2026-05-05 | **Path-scheme migration.** Inputs repointed to `2-scenarios/pdd-to-app-journeys.md`, `3-commcare/app-test-cases.yaml`, `6-qa-and-training/screenshots/` + `6-qa-and-training/app-screenshot-capture_manifest.yaml`, `3-commcare/app-deploy_summary.md`. Verdict output is now `6-qa-and-training/app-ux-eval_verdict-deep.yaml` (per manifest). Calibration audit trail at `ACE/<opp>/eval-calibration/app-ux-eval-runs.md` corrected to opp-level (was incorrectly under `runs/<run-id>/`). Phase references updated 6 → 7 to match the 8-phase topology. No behavior change beyond paths. | ACE team |
| 2026-05-31 | **Meaningful journey ids.** `per_item[].ref`/`journey` keys now use the journey's meaningful slug id (`learn-happy-path`, `deliver-yes`) from `app-test-cases.yaml` instead of `J<n>` ordinals; example verdict updated. See `skills/app-test-cases/SKILL.md § Journey id convention`. | ACE team |
| 2026-05-31 | **`journey-` prefix.** `per_item[].ref`/`journey` example values now carry the `journey-` prefix (`journey-deliver-submit`, `journey-deliver-alt-answer`) to match the amended id convention. See `skills/app-test-cases/SKILL.md § Journey id convention`. | ACE team |
| 2026-05-29 | **Fitness dim + time_budget fix (ITN post-mortem).** Added `capture_robustness` (0.30) — grades negative-path journeys (blank-required / out-of-range / low-GPS / "Other") for whether the form *refuses bad data*, with an adversarial-coverage cap (zero negative-path journeys → ≤2, not pass). Fixed `time_budget`: being far UNDER budget is now a thinness WARN (≤2), not a 9.5 — the old rule only penalized "too slow," which rewarded the ITN-style skeletal build. Reweighted to 6 dims (capture_robustness heaviest at 0.30). Per `_eval-template.md § out-of-chain fitness requirement` + `docs/superpowers/specs/2026-05-29-eval-fitness-gap.md`. Note: in-`/ace:run` build-fitness gating is handled cheaply (no AVD) by the revised `pdd-to-*-app-eval` blueprint checks; this deep screenshot check is complementary. | ACE team |
| 2026-09-29 | **Frames resolved by `file_id`, not by folder (output previews contract v1).** Smoke frames moved to `3-commcare/previews/<app-output-slug>/`; deep frames stay in `6-qa-and-training/screenshots/<recipe-base>/`. The capture manifest is the only reference. | ACE team |
| 2026-10-05 | **NOT REACHED criteria (dimagi-internal/ace#2670).** Per-journey `not_reached: [{criterion, reason}]` plus a flattened top-level `not_reached`. Each NOT REACHED criterion is a `[WARN]` (owner INSTRUMENT) and caps that journey's `journey_completion` at 2. It is never the didn't-finish hard deduction and never a pass. The gate summary names every one. Earned by `spark-facilitator/20261004-1706`, where two `reachability: multi-day` criteria (N dated meetings on one case under a strictly-increasing `<= today()` validate) could not be reached in one session, and the rubric had no tier for them. | ACE team |
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall