Skip to content

hoangsonww/Claude-Code-Agent-Monitor / plugins/ccam-insights/skills/regression-watch

regression-watch

Verified

Detect quality and efficiency regressions over time using Agent Monitor data — rising error rate (APIError events), falling cache hit rate, growing compaction frequency, and climbing cost-per-session. Splits history into an earlier baseline window and a recent window and reports which metrics are getting worse, by how much, and where. Use when checking whether things are degrading or trending in the wrong direction.

FileKindSize
SKILL.mdskill md4.0 KB
agents/openai.yamlconfig270 B
---
name: regression-watch
description: >
  Detect quality and efficiency regressions over time using Agent Monitor data —
  rising error rate (APIError events), falling cache hit rate, growing compaction
  frequency, and climbing cost-per-session. Splits history into an earlier
  baseline window and a recent window and reports which metrics are getting
  worse, by how much, and where. Use when checking whether things are degrading
  or trending in the wrong direction.
---

# Regression Watch

Detect whether Claude Code sessions are getting worse over time across quality and
efficiency metrics, using Agent Monitor data.

## Input

The user provides: **$ARGUMENTS**

This may be:
- empty or "all" — check every regression metric (default)
- "errors" — error-rate regression only
- "cache" — cache hit-rate regression only
- "compaction" — compaction-frequency regression only
- "cost" — cost-per-session regression only
- A window like "last 30d" or "30 vs 90" — set the recent vs baseline window sizes

## Data Sources

| Endpoint | Returns |
|----------|---------|
| `GET /api/analytics` | `daily_events` (365d), `daily_sessions` (365d), `event_types`, `tokens` (total_input, total_output, total_cache_read, total_cache_write — baselines pre-summed), `avg_events_per_session` |
| `GET /api/events?session_id=X` | Event stream incl. `APIError`, `Compaction`, `PreToolUse`/`PostToolUse` — used to localize regressions to specific sessions |
| `GET /api/pricing/cost` | `{ total_cost, breakdown[...] }` — total cost to derive cost-per-session |
| `GET /api/pricing/cost/{sessionId}` | Per-session cost — used to compare recent vs baseline session cost |
| `GET /api/workflows/{sessionId}` | `compaction` (impact), `errorPropagation` (by depth), `effectiveness` — per-session quality signals |
| `GET /api/sessions?limit=N` | Sessions with `started_at`, `cost`, `metadata` — to bucket sessions into time windows |

## Report Sections

### 1. Windowing
Split history into a **baseline window** (older) and a **recent window** (newer).
Default: recent = last 30 days, baseline = the 30–90 day range before it. Use
`daily_events`/`daily_sessions` for series metrics and `GET /api/sessions?limit=N`
to assign sessions to each window by `started_at`.

### 2. Error Rate Regression
- Recent error rate = `APIError count / total events` in the recent window
  (from `event_types` and `daily_events`, or per-session `GET /api/events`).
- Compare to the baseline rate. Flag if recent is higher.
- Report the absolute and relative change and which sessions contributed most
  `APIError` events.

### 3. Cache Hit Rate Regression
- Cache hit rate = `total_cache_read / (total_cache_read + total_input)`.
- Compute for each window (per-window input/cache_read from session metadata or
  the pricing breakdown). Flag a **falling** hit rate — that means more
  uncached input tokens and higher cost.

### 4. Compaction Frequency Regression
- Compaction frequency = `Compaction events / session` per window (from
  `event_types` / `daily_events`, confirmed via per-session
  `GET /api/workflows/{id}` `compaction`). Flag a **rising** rate — context is
  overflowing more often.

### 5. Cost-per-Session Regression
- Cost-per-session = window total cost / window session count, using
  `GET /api/pricing/cost` overall and `GET /api/pricing/cost/{id}` for the
  sessions in each window. Flag a **climbing** value.

### 6. Verdict
Roll up which metrics regressed, rank by relative worsening, and name the most
likely driver (e.g., cache hit rate fell → cost per session climbed).

## Output

- A Markdown table: metric | baseline | recent | Δ | direction (▲ worse / ▼ better) | verdict.
- Tag each regressed metric 🔴 (clear regression), 🟡 (mild/within noise), or 🟢 (improved).
- Currency in USD to 4 decimals; rates as percentages to 2 decimals.
- List the specific session IDs that contributed most to any regression.
- End with the single highest-priority regression to address and a concrete next step.
- Read-only: only report what the API returns; never fabricate baselines.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall