Skip to content

borghei/Claude-Skills / project-management/execution/agents-in-the-team

agents-in-the-team

Unverified

Run delivery when AI coding and ops agents take tickets. Use when setting agent delegation policy, writing agent-ready tickets, planning review capacity, measuring agent vs human delivery, or rolling agents out safely.

references/metrics-and-capacity.md

View on GitHub
# Metrics and Review Capacity

Deep reference for measuring delivery when agents author part of the work, and for planning around the new bottleneck: human review.

---

## 1. What DORA says about AI (2025)

DORA's 2025 research report, *State of AI-assisted Software Development*, and Google Cloud's announcement of it state the following. Quote these, do not paraphrase them into stronger claims:

- AI's primary role is as an **amplifier**, magnifying an organization's existing strengths and weaknesses.
- **90%** of survey respondents report using AI at work; **more than 80%** believe it has increased their productivity; **30%** report little or no trust in the code generated by AI.
- "We observe a positive relationship between AI adoption on both software delivery throughput and product performance." However, "AI adoption does continue to have a negative relationship with software delivery stability."
- DORA's reading: AI accelerates development, but that acceleration can expose weaknesses downstream; without robust control systems (strong automated testing, mature version control, fast feedback loops) more change volume leads to instability.

**Implication for a team adding agents:** expect throughput to rise and plan explicitly for stability. The metrics below exist to see instability early, split by who authored the change.

### The DORA AI Capabilities Model

Google Cloud's guide to the DORA AI Capabilities Model names seven capabilities that amplify the benefit of AI:

1. Clear and communicated AI stance
2. Healthy data ecosystems
3. AI-accessible internal data
4. Strong version control practices
5. Working in small batches
6. User-centric focus
7. Quality internal platforms

How this skill maps to them:

| Capability | Where it shows up here |
|------------|------------------------|
| Clear and communicated AI stance | The delegation policy (`assets/delegation_policy_template.md`) |
| AI-accessible internal data | Ticket context links and patterns; repository instructions for the agent |
| Strong version control practices | Agent works on its own branches; branch protection; easy revert |
| Working in small batches | Ticket scope limits; split large agent diffs |
| User-centric focus | Acceptance criteria stated as user/system outcomes |
| Quality internal platforms | CI, test suites and environments that let reviewers trust green checks |
| Healthy data ecosystems | Out of scope for this skill (data platform work) |

---

## 2. DORA software delivery metrics

DORA defines five software delivery performance metrics (dora.dev), in two groups:

| Group | Metric | DORA definition |
|-------|--------|-----------------|
| Throughput | Change lead time | Time for a change to go from committed to version control to deployed in production |
| Throughput | Deployment frequency | Number of deployments over a period, or time between deployments |
| Throughput | Failed deployment recovery time | Time to recover from a deployment that fails and requires immediate intervention |
| Instability | Change fail rate | Ratio of deployments that require immediate intervention following a deployment |
| Instability | Deployment rework rate | Ratio of deployments that are unplanned but happen as a result of an incident in production |

`agent_delivery_metrics.py` computes change fail rate, deployment rework rate and median failed deployment recovery time, **split by deployments that contained any agent-authored PR vs human-only deployments**. Lead time and deployment frequency are team-level and are not meaningfully split by author; use your existing pipeline data for them.

**Attribution caveat:** a deployment that contains both agent and human changes is counted in the "with agent changes" group. That is deliberately conservative. If most deployments are mixed, the split loses resolution: deploy smaller batches, which is DORA's own advice.

---

## 3. Agent-specific metrics

| Metric | Definition in the tool | What it tells you | Watch for |
|--------|------------------------|-------------------|-----------|
| **PR acceptance rate** | merged / (merged + closed unmerged) | How often agent work is worth keeping | Low rate = ticket quality or eligibility problem, not an agent problem |
| **Review rework rate** | merged PRs with 1+ "changes requested" / merged | How often review had to correct the work | High and stable = reviewers are doing the agent's thinking |
| **Post-merge rework rate** | merged PRs later reverted or referenced by a `fix_of` PR / merged | Defects that got past review | The most important quality signal |
| **Median hours to first review** | opened to first review | Queueing: is review the bottleneck? | Rising while agent PR count rises = capacity problem |
| **Median hours open to merge** | opened to merged | End-to-end PR cycle | Compare cohorts, not absolute values |
| **Median reviewer minutes per PR** | from `review_minutes` | Human cost of each PR | Agent PRs that take longer to review than human PRs erase the speed gain |
| **Escaped defects per 10 merged** | production defects with `caused_by_pr` in the cohort | Customer-visible quality | Any sustained gap vs human cohort |

All ratios are shown per cohort (human vs agent). Cohorts under 10 merged PRs are flagged as small samples: read them as directional.

### The gate [RECOMMENDED]

`--gate --tolerance 0.05` exits 2 when the agent cohort's post-merge rework rate, change fail rate or escaped defects per merged PR exceeds the human cohort's by more than 5 percentage points. The tolerance is a team policy choice, not an industry benchmark. Set it in the rollout plan and use a breach as a trigger to narrow the delegation bands.

### What not to measure

- **Lines of code or PR count as productivity.** The Scrum Guide Expansion Pack's AI expansion warns that AI makes outputs easy to measure and asks teams to remain outcome-driven.
- **Agent "utilization".** Idle agents cost little; overloaded reviewers cost a lot.
- **Individual reviewer leaderboards.** They reward rubber-stamping.

---

## 4. Review capacity: the new bottleneck

When an agent can open PRs faster than humans can review them, **reviewer hours become the constraint on throughput**. Plan them like any other capacity.

### The capacity check in `agent_delegation_scorer.py`

```
review_hours(ticket) = (base_hours + hours_per_point x points) x risk_multiplier
                       x (1 + rework_allowance)          # or est_review_hours if given
planned_load        = sum(review_hours for tickets planned for an agent)
budget              = sum(reviewer hours this sprint) x max_review_utilization
gate                = planned_load <= budget
```

Defaults: `base_hours` 0.5, `hours_per_point` 0.5, risk multiplier 1.0 / 1.5 / 2.0, `rework_allowance` 0.3, `max_review_utilization` 0.8. **These are planning assumptions, not measured industry values.** Replace them with your own numbers after two sprints:

- `rework_allowance` = your agent **review rework rate** from `agent_delivery_metrics.py`
- `hours_per_point` = median reviewer minutes per agent PR / 60, divided by median points
- `max_review_utilization` < 1.0 because reviewers also review human PRs, handle interrupts and write code

### When the gate fails

Cut the plan, not the review. In order of preference:

1. Drop the lowest-scoring agent tickets (agent-assisted band first).
2. Move large agent tickets to next sprint or split them.
3. Add reviewer hours deliberately (a named person, with other work removed).
4. Never: lower review depth, auto-approve, or let the requester self-approve.

---

## 5. Sources

- DORA, *DORA's software delivery performance metrics*: https://dora.dev/guides/dora-metrics/
- DORA, *State of AI-assisted Software Development 2025*: https://dora.dev/dora-report-2025/
- Google Cloud Blog, *Announcing the 2025 DORA Report*: https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
- Google Cloud Blog, *From adoption to impact: Putting the DORA AI Capabilities Model to work*: https://cloud.google.com/blog/products/ai-machine-learning/from-adoption-to-impact-putting-the-dora-ai-capabilities-model-to-work
- Scrum Guide Expansion Pack, *AI and Scrum*: https://scrumexpansion.org/ai-and-scrum/
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall