Skip to content

dimagi-internal/ace / skills/simulated-execution-qa

simulated-execution-qa

Unverified

Pass/fail check of a simulated N-week execution: calendar, weekly runs, isolation, no LLO contact. Static.

Simulated execution QA

Structural checks on a whole simulated Phase 9, judged against what labs holds now and what the run recorded. Each week's stock ledger is supply-chain-operate-qa's job and each review's numbers are simulated-weekly-review-eval's; this checks that the weeks add up to a simulation, and that it stayed a simulation.

See skills/_qa-template.md for the shared QA contract.

Inputs

ParameterUsed forResolver
calendarthe weeks judgedexplicit arg; default: run_state phases.execution-management.products.simulation.calendar
definition_id, synthetic_opportunity_idthe weekly report and its scopeexplicit arg; default: run_state products.simulation.{workflow_id, synthetic_opportunity_id}
enabledsupply_chain, simulate_weeksexplicit arg; default: run_state optional_steps
selected_llothe handover fields that must stay emptyexplicit arg; default: run_state phases.solicitation-management.products.selected_llo
winner_org_namethe org the labs-only opportunity must be filed underexplicit arg; default: connect_orgs.llo_org's display name
out_folder_idwhere the per-week QA results and reviews areexplicit arg; default: the run's 9-execution-manager/
supply_program_id, visit_program_idcheck 9run_state products.supply_chain.program_id, products.simulation.program_id
supply_motioncheck 109-execution-manager/simulated-execution_supply-motion.yaml (simulated-execution Step 8)
reportscheck 11workflow_list({opportunity_id: <synthetic opp>}) mapped by template_type (indicator_programme_report → programme, indicator_opp_report → partner, indicator_worker_review → worker), each with its run count from workflow_history_runs

Products

  • 9-execution-manager/simulated-execution-qa_result.yaml, built with aggregateQAResult (lib/qa-types.ts).

Checks

The functions live in skills/simulated-execution-qa/checks.ts. Each takes one JSON document, SimulationDoc.

#idtypedescriptionauto-fix on fail
1calendar_backdatedstaticMonday-aligned, contiguous, all before the run start, N = simulate_weeksrecompute with scripts/sim-calendar.ts
2week0_supply_chainstaticprocure and fulfil QA passed (supply_chain only)finish week 0 first
3every_week_operatedstaticevery delivery week's supply-chain-operate-qa passed (supply_chain only)re-run the week; it replays by source.ref
4history_one_run_per_weekstaticexactly one completed saved run per delivery week, on its Monday-Sunday periodworkflow_rebuild_history over the window
5synthetic_opportunity_isolatedstaticvisits on a labs-only opportunity (id ≥ 10000, a labs-synthetic- org) whose org name is the ACE winner'screate a labs-only opportunity (synthetic_create_labs_only), generate into it, repoint the report
6review_per_weekstatica weekly review per delivery weekrun simulated-weekly-review
7no_real_llo_contactstaticsynthetic_winner: true, no real handover field set, no Phase 9 sendstop and report: a real step ran
8summary_links_every_weekstaticthe summary links every reviewrewrite the summary
9one_programmestaticthe supply chain and the visits are on one labs-only synthetic programme (supply_chain only)rebuild on products.simulation.program_id; records elsewhere cannot be moved
10supply_in_motionstaticassessSupplyMotion over labs' reads: stock leaves the central store in at least half the weeks and covers what was handed out (not a share of the order, which is sized for the whole programme), every worker gave out at visits labs read, workers tied to the opportunity, at least half ever hold stock at a week end, stores with coordinates (supply_chain only)the verdict's problems name the cause, in order
11downstream_reportsstatica programme, a partner and a worker report read the synthetic opportunity, each with a saved runclone and repoint per simulated-execution Step 5

Checks 9-11 are the owner's bar (ace#3265): connect-fi/20261018-1440 passed checks 1-8 while its labs pages showed a supply chain standing still. Their negative controls are that run's live reads.

Process

  1. Read fresh: workflow_history_runs({definition_id, opportunity_id: <synthetic opp>, generated_only: true}) (page with start_at until done); labs_context({search: <winner org name>}) for the synthetic opportunity's org (slug and name; labs carries no labs_only flag, so the labs-synthetic- slug and the id range say it); each week's supply-chain-operate-qa_week-<w>_result.yaml, the week-0 QA results and the review files from the folder; run_state's selected_llo and optional_steps; Phase 9's outbound sends from the run's comms-log.
  2. Build SimulationDoc and run CHECKS over JSON.stringify(doc); write the result with aggregateQAResult.
  3. On fail, return the failures and hints. no_real_llo_contact failing is never auto-fixed: it means the simulated carve-out was breached, and the run stops for the operator.

MCP Tools Used

  • connect-labs: workflow_history_runs, labs_context, workflow_list
  • ace-gdrive: drive_list_folder, drive_read_file, drive_create_file

Change Log

DateChangeAuthor
2026-10-21Checks 9-11 (one_programme, supply_in_motion, downstream_reports): the gate is what a viewer sees; negative controls on connect-fi/20261018-1440's live reads (ace#3265)ACE
2026-10-19Check 5 wording: the labs-only opportunity is created by synthetic_create_labs_only, not cloned (ace#3263); observed live: org labs-synthetic-ace-test-llo--synthetic, "ACE Test LLO (Synthetic)", id 10168ACE
2026-10-18Initial version (optional-track PR10): 8 static checks; the weekly-history control is grounded on a live capture of labs workflow 7410's rebuilt historyACE
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 318kUpdated Claude CodeCodexUnverifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 318kUpdated Claude CodeCodexUnverifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 318kUpdated Claude CodeCodexUnverifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall