Adonis0123/adonis-skills / skills/dev-environment-roi-audit
dev-environment-roi-audit
VerifiedAudit a local coding-agent environment with real logs and controlled ablations, then make evidence-backed keep, narrow, on-demand, or remove decisions for MCP servers, skills, instruction files, plugins, commands, hooks, and memory, plus the attention they cost the user (notifications, pending decisions, scheduled automations, and settings toggles). Use whenever the user asks to audit or slim a Claude, Codex, Grok, Cursor, or similar development setup; find unused tools; quantify context, startup, or attention tax; review noisy notifications or scheduled jobs; or clean agent configuration by ROI. Enforces discovered paths, per-file JSONL schema probes, OUT_DIR-only artifacts, secret-safe output, and item-by-item approval before mutations.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.3 KB |
| evals/evals.json | config | 2.3 KB |
| evals/fixtures/schema-a.jsonl | other | 72 B |
| evals/fixtures/schema-b.jsonl | other | 151 B |
| references/artifact-contract.md | markdown | 2.2 KB |
| references/metrics-and-decisions.md | markdown | 5.5 KB |
| scripts/finalize_audit.pyexecutable | script | 2.4 KB |
| scripts/init_audit.pyexecutable | script | 2.0 KB |
| scripts/probe_jsonl_schema.pyexecutable | script | 2.6 KB |
references/metrics-and-decisions.md
View on GitHub# Metrics and decisions
## Window
Use the most recent 90 days by default. Report:
- requested window
- actual earliest/latest timestamp
- session count per host
- files parsed, skipped, and schema-mismatched
If timestamps are absent, do not silently substitute mtime. Mark mtime-based windowing as a proxy.
## M1-M11
### M1 — Calls by tool
Count structured tool invocations. Keep host-qualified tool identities. Report current exposed tools with zero observed calls separately.
### M2 — Error rate
```text
error_rate = failed_calls / total_calls
```
List tools above 20% with sample sanitized errors. A wrapper returning successfully does not prove the nested MCP call succeeded.
### M3 — Server usage rate
```text
usage_rate = distinct_called_exposed_tools / exposed_tools
```
Use the currently exposed tool set. Label historical tool sets separately.
### M4 — Server ROI
```text
resident_tokens = measured_ablation_delta
fallback_resident_tokens = total_schema_bytes / 4
ROI = total_calls / (resident_tokens / 1000)
```
The byte/4 fallback is a lower-confidence estimate, not an empirical token measurement. Rank all servers and state the cost method.
### M5 — Skill utility
Count explicit structured triggers where available. Inspect the following messages/tool actions to decide whether the body affected execution. Separate trigger count from useful-trigger count.
### M6 — Session pressure
Calculate token/session p50 and p95, plus context-compaction sessions / total sessions. Do not mix input, output, cache creation, and cached-input fields without documenting the formula.
### M7 — Rework
Measure repeated edits to the same file within a session and retry-loop lengths after consecutive failures. Do not infer rework from repeated reads.
### M8 — Latency
Calculate p50/p95 only when start/end timestamps define the same boundary. Otherwise write `UNVERIFIED`.
### M9 — Attention load
Per notification sender and per decision queue:
```text
interruptions_per_week = messages_or_new_items / window_weeks
action_rate = interruptions_followed_by_user_action / interruptions
```
A user action is a human message, approval, or command in the same session or queue within 30 minutes; state the window you used. For decision queues also report item age p50/p95, the duplicate share (same question recorded twice), and the share closed without user input (done by an agent, superseded, or expired). Without a sender log, write `UNVERIFIED`; do not estimate from memory.
### M10 — Automation reliability
Per scheduled automation:
```text
failure_rate = runs_with_nonzero_exit / runs_in_window
```
Also count runs that changed user state the user had not approved, such as overwriting local edits or reinstalling a removed component. Take run counts from logs, not from the schedule; a missed run is a failure of its own kind and is listed separately.
### M11 — Toggle drift
```text
static_toggles = toggles_unchanged_in_window / toggles
```
List each unchanged toggle with whether it still equals the product default. A toggle that differs from the default is a decision the user made; it is not static waste.
## Decisions
Apply these hard rules first:
- **Remove**: zero calls in the measured window.
- **Remove**: resident cost above 2,000 tokens and fewer than 5 calls.
- **Narrow**: the component has value, but the exposed tool set greatly exceeds the called subset. Flag usage rate at or below 25% for explicit review.
- **On demand**: value is demonstrated, but startup/resident cost need not be paid every session.
- **Rewrite**: a skill or instruction file is used and worth its cost, but its error rate (M2) or rework (M7) points at its own text. Draft two variants: a procedure variant (planning, tool use, completion, context cost; no subject matter) and a domain variant (facts and constraints a practitioner would know). Each variant emits only what it changes. Replay the same redacted prompts against the current text and both variants, then judge anonymized, shuffled outputs against the source material. Adopt, merge, or reject each change only by a behavior difference observed in those runs; when no run shows a difference, keep the current text.
- **Digest**: a notification sender has an action rate below 20% over at least 10 interruptions. Fold its messages into a periodic summary, or drop the event type, instead of interrupting.
- **Close duplicates**: a decision queue whose duplicate or agent-closable share exceeds 30%. Propose closing those items and fixing the recorder so one question is stored once.
- **Observe first**: a scheduled automation in `acts` mode that failed in the window or changed user state without approval. Switch it to `reports` (dry run, or skip the affected items and list them) until it has 4 consecutive clean runs, then offer `acts` again.
- **Use the default**: a toggle unchanged in the window that equals the product default. Propose removing the explicit setting. Keep it when a comment or record gives a reason.
- **Keep**: measured ROI is positive and no harder remove rule applies.
User decisions can override the recommendation. Record both the threshold result and the user's decision; never rewrite the metric to match the preference.
## Baseline bill
For each host, report:
```text
floor = bare or ignore-user-config minimal prompt
full = identical prompt with current config
fixed_tax = full - floor
window_share = fixed_tax / context_window
```
Only call the difference causal when the prompt, model, account, cwd, timing, and all non-tested config are held constant.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |