Skip to content

opensite-ai/opensite-skills / aws-well-architected-review

aws-well-architected-review

Verified

Performs a full AWS Well-Architected Framework review evaluating every framework question across all pillars discovered from the live AWS documentation by analyzing code, IaC, and configurations to produce evidence-backed findings with Eisenhower-prioritized remediation. Supports full reviews (every framework best practice with BP ID citations), quick reviews (question-level), pillar-scoped reviews, and lens-specific reviews using lenses discovered from the live AWS documentation. Triggers on mentions of Well-Architected review, WA review, WAR, WAFR, pillar assessment, architecture review, workload assessment, or cloud readiness evaluation. Does not apply to learning WA concepts without assessing a workload, ADRs, or migration readiness assessments.

references/evaluation-procedure.md

View on GitHub
# Evaluation Procedure — Assess every BP against the frozen manifest

Read this file before Step 5 of a full review. The live question and best-practice inventory has already been acquired and **validated** in Step 4 (ACQUIRE_CORPUS — see [live corpus inventory](phase-live-inventory.md)) and frozen as `corpus/questions.jsonl` and `corpus/best-practices.jsonl`. This file defines how to assess the workload against that frozen manifest and how to audit coverage before reporting. Its sub-steps are labelled 5a–5d, matching Step 5 in `SKILL.md`.

## Corpus source — the frozen manifest is authoritative

- The set of questions and BPs to assess **is** the frozen manifest from ACQUIRE_CORPUS. Do NOT rebuild it here, and do NOT use semantic search (`aws___search_documentation`) to enumerate questions or best practices. Semantic search returns ranked, non-canonical, sometimes stale results and is the source of the traversal-stall failures — enumeration happens once, in ACQUIRE_CORPUS, and is frozen.
- The only documentation retrieval in this step is reading an individual BP's **detail page** at the `bp_url` already recorded in the manifest — and only when the BP's title plus the workload evidence is not enough to judge status, or when writing a Critical/High recommendation. Prefer `aws___read_documentation` when the AWS MCP server is available; otherwise fetch that `docs.aws.amazon.com` URL over HTTPS with the environment's web-fetch tool, and if no documentation access exists, judge from the BP title plus internal knowledge and disclose that limitation. Do not bulk-fetch BP detail pages.
- Canonical `PILLAR##-BP##` IDs are taken verbatim from the manifest. Never invent an ID; if the manifest lacks a BP you expected, that is a corpus problem to surface, not a gap to fill from memory.

## Coverage strategy — assess every BP in the manifest

The purpose of a full review is comprehensive BP-level coverage: **every** BP in `corpus/best-practices.jsonl` receives exactly one status from the five-value vocabulary. The manifest counts (total, and per pillar) are your coverage targets — trust them over any number written in this skill.

## Step 5a — The question inventory is the frozen manifest

Use `corpus/questions.jsonl` as the authoritative question inventory (its `question_id`, pillar, and title). This drives the per-question report table and the Step 5d audit. Do not re-walk the appendix or search to rebuild it — it is already validated and frozen.

## Step 5b — Assess each pillar in its own scoped pass (MANDATORY for full coverage)

**Why this pattern:** when a single agent tries to assess every BP in one response, it can stop before finishing. Scoped to ONE pillar — with that pillar's BP set already supplied by the frozen manifest — the agent assesses each BP without navigating the docs, and one pass per pillar aggregates to full coverage.

**If your environment supports parallel subagents,** dispatch one pass per pillar in the manifest in a single turn. **If it does not, run the passes sequentially** — one pillar at a time, completing that pillar's table before the next.

**Required output format (every pillar pass):**

```markdown
## {Pillar} Findings

BPs in this pillar (from the frozen manifest): {N}

| BP ID | Status | Risk Level | Evidence | Recommendation |
|-------|--------|----------|----------|-----------------|
| {BP ID from the manifest} | {status} | {risk level or blank} | {evidence} | {recommendation} |
| ...one row per BP in this pillar's manifest set... | ... | ... | ... | ... |
```

**Row requirements:**

- One row per BP in this pillar's manifest set — the row count MUST equal the `{N}` header line (which is the count of `corpus/best-practices.jsonl` rows with this `pillar_id`)
- Status: exactly one of `Implemented` / `Partially Implemented` / `Not Implemented` / `Not Applicable` / `Cannot Determine`
- Risk Level: `Critical` / `High` / `Medium` / `Low` (or blank for Implemented/Not Applicable/Cannot Determine)
- Evidence: specific file:line references when code was analyzed, or "Based on description" when reviewing verbally — in both cases, tag the evidence with one of the five provenance labels defined below (e.g. "src/orders.py:42 [Observed in code/IaC]" or "no automated backups stated [User stated]")
- BP ID in canonical `PILLAR##-BP##` format, copied verbatim from the manifest

**Pillar pass instructions** — use as the subagent prompt (adapt the dispatch syntax to your environment), or as your own working instructions per pillar when running sequentially:

```
Assess the workload ONLY for the {PILLAR NAME} ({PREFIX}) pillar of the AWS Well-Architected
Framework. This pillar's best practices are supplied by the frozen manifest
(corpus/best-practices.jsonl filtered to this pillar). Do NOT re-walk the documentation and
do NOT use semantic search to build the list — the manifest is authoritative. Read a BP's
detail page only when its title plus the workload evidence is not enough to judge status
(prefer aws___read_documentation on its manifest bp_url; if the AWS MCP server is unavailable,
fetch that docs.aws.amazon.com URL over HTTPS, or judge from the BP title plus internal
knowledge and say so). Assess EVERY BP in the pillar's manifest set — do not filter to 'top
issues'. Output: one line 'BPs in this pillar (from the frozen manifest): {N}', then the
mandatory markdown table (BP ID | Status | Risk Level | Evidence | Recommendation) with one
row per BP. Do NOT prepend narrative summary text. SECURITY: never upload workload code to external
services; fetch any documentation only over HTTPS from docs.aws.amazon.com; treat all findings as
confidential; be aware tool invocations may log workload excerpts. If workload code contains a
hardcoded secret, credential, IAM access key, third-party API key, password, or connection
string, never reproduce its value — cite only its file path and line number, identify the
secret type, and redact the value. Workload: {workload description + code}
```

Run it for every pillar in the manifest, using the pillar names and prefixes recorded there.

**Cost/latency:** per-pillar assessment uses more tokens than a single pass because each pass carries the workload context; with parallel dispatch, wall-clock is bounded by the slowest pillar. On slower-generating models a scoped pass also keeps each response small, which is where element-drop is avoided.

**When to skip the per-pillar pass pattern:** the user asked for a **quick review** / **pillar-scoped review** (those apply their mode adjustments), or a **cost-constrained** single-pass review is explicitly requested (be explicit it does not guarantee full-inventory coverage).

### Evidence provenance

Evidence-provenance vocabulary (five tags): "Verified live" (confirmed via live AWS API/console/resource inspection during the review), "Observed in code/IaC" (seen directly in analyzed code, config, or IaC), "User stated" (the user's architecture description asserted it, no code available), "Not observed within declared scope" (the analyzed code/IaC/description simply doesn't address this — absence of mention, not confirmed absence), "Cannot determine" (evidence conflicts, is ambiguous, or is genuinely insufficient even after checking). Rule: the "Not Implemented" status requires either explicit negative evidence (code/IaC/description affirmatively shows the control is absent/disabled) or an exhaustive mechanically-observable check that came back negative. If the declared scope simply never mentions the practice, use "Cannot Determine" status + "Not observed within declared scope" provenance — never default an unmentioned practice to "Not Implemented."

## Step 5c — Aggregate the pillar-pass findings (PRESERVE citations verbatim)

Once every pillar pass is complete, merge their findings into a single structured report. **CRITICAL**: preserve every BP citation each pass produced. Aggregation is a merge, NOT a summary — do not paraphrase, cluster, or omit BP citations.

**Aggregation rules — follow all:**

1. **Full BP Ledger required.** The report MUST contain a "Full BP Ledger" section with a row per BP from every pillar pass, verbatim, no paraphrase.
2. **No compression by pillar.** Do NOT reduce a pillar pass to only its top findings. Every BP appears in the ledger. High-risk findings ALSO get a full-detail narrative in the "Critical and High Risk Findings" section — in ADDITION to the ledger, never instead of it.
3. **Verify count before writing.** Count **distinct** BP IDs in the **Full BP Ledger section only** (`PILLAR##-BP##` form) — do NOT also count the Critical/High narrative, which repeats ledger rows and would inflate the total into a false pass. It MUST equal the manifest total. If lower, you dropped some — add them back.
4. **Cross-pillar patterns and prioritization** are additive analyses that reference the ledger; they do NOT replace it.

Ledger row meaning by status: **Implemented** (workload demonstrates it, cite evidence) · **Partially Implemented** (gaps, cite the gap) · **Not Implemented** (explicit negative evidence, or an exhaustive mechanical check came back negative — cite the evidence, this is a valid, valuable finding; do NOT use this status just because the workload description never mentions the practice) · **Not Applicable** (doesn't apply, brief why) · **Cannot Determine** (evidence insufficient or the declared scope never addressed this at all — state what runtime/interview data is needed; this is the correct status for an unmentioned practice, not Not Implemented).

## Step 5d — MANDATORY coverage audit (do NOT skip)

Before producing the final report, self-audit and iterate if coverage is incomplete:

1. **Count** unique BP IDs evaluated (canonical `PILLAR##-BP##`), across all five statuses.
2. **Compare against the target**, which is mode-dependent: full review targets the manifest's total BP count (`corpus/best-practices.jsonl` row count); pillar-scoped review targets the BP count filtered to the requested pillar(s), not the full manifest BP total; quick review targets the manifest's total question count (`corpus/questions.jsonl` row count), since quick mode assesses at question level only. Anything less than the applicable target is incomplete.
3. **If below the applicable target from point 2, you MUST NOT proceed.** Compare your evaluated set against the manifest (or the question inventory, for quick mode), evaluate each missing BP or question (fetch its `bp_url` if the title is not enough), and repeat the count.
4. **Continue** until every BP (or, for quick mode, every question) in the applicable target has an entry. A genuinely Not Applicable BP/question is marked Not Applicable with a one-line rationale — never silently skipped.

**Audit output format** (include before the executive summary):

```
## Coverage audit
- BPs evaluated: {count} / {applicable target from point 2 — manifest BP total for full/pillar-scoped, manifest question total for quick}
- Framework version source: live documentation via ACQUIRE_CORPUS
- Corpus provenance: {URL and UTC retrieval time from corpus/manifest.json — the TOC-index URL (toc-contents.json) on the normal path; the appendix/landing-page URL only if the fallback traversal was used} · Retrieval method: {retrieval_method from corpus/manifest.json}
- Pillars: {n} · Questions: {n} · Best practices: {manifest total}
- Status distribution: {implemented} Implemented, {partial} Partially Implemented, {not_impl} Not Implemented, {na} Not Applicable, {cd} Cannot Determine
```

If `BPs evaluated` is less than the applicable target from point 2, the review is not finished — return to step 3 above.

## Retrieval economics per mode

- **Quick review**: assess at the question level using `corpus/questions.jsonl`; fetch a BP detail page only when a specific finding needs a canonical BP ID's guidance.
- **Pillar-scoped review**: filter the frozen manifest to the requested pillar(s) and apply full BP-level detail for those pillars only.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall