Skip to content

FrancyJGLisboa/agent-skills-platform / references/examples/pr-blocker-summarizer

pr-blocker-summarizer

Unverified

Summarizes open pull requests into a blockers-first standup digest. Activates when the user asks to summarize open PRs, find blocked pull requests, generate a PR standup, or triage review backlog from a PR export.

evals/pr-blocker-summarizer.eval.md

View on GitHub
# Eval Spec: pr-blocker-summarizer

The skill's loss function: structural binary checks plus golden PR exports. Run
`python3 scripts/run_evals.py` or, end-to-end,
`python3 scripts/run_evals.py --rollout`.

## Criteria

1. **valid-json** (command) — the digest parses as JSON.
2. **has-summary-line** (command) — a non-empty `summary` string is present.
3. **counts-consistent** (command) — `blocked` + `ready` counts do not exceed `total`.

## Golden cases

Three PR exports spanning blocked, ready, and stale states. Cases are
`pending-first-green`; capture a baseline with `--rollout --promote`.

## Spec

```json
{
  "skill": "pr-blocker-summarizer",
  "run": "python3 scripts/run_pipeline.py --input {input} --output {output}",
  "criteria": [
    {"id": "valid-json", "text": "Digest parses as JSON", "type": "command", "cmd": "python3 -c \"import json,sys; json.load(open(sys.argv[1]))\" {output}"},
    {"id": "has-summary-line", "text": "Non-empty summary string", "type": "command", "cmd": "python3 -c \"import json,sys; d=json.load(open(sys.argv[1])); assert isinstance(d['summary'],str) and d['summary']\" {output}"},
    {"id": "counts-consistent", "text": "blocked+ready <= total", "type": "command", "cmd": "python3 -c \"import json,sys; d=json.load(open(sys.argv[1])); assert len(d['blocked'])+len(d['ready']) <= d['total']\" {output}"}
  ],
  "golden": [
    {"id": "case-1", "input": "golden/case-1/input.json", "expected": null, "split": "val", "expected_status": "pending-first-green"},
    {"id": "case-2", "input": "golden/case-2/input.json", "expected": null, "split": "test", "expected_status": "pending-first-green"},
    {"id": "case-3", "input": "golden/case-3/input.json", "expected": null, "split": "val", "expected_status": "pending-first-green"}
  ]
}
```
add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.Stars: 115kUpdated Claude CodeCodexVerifiedInstall
fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.Stars: 115kUpdated Claude CodeCodexVerifiedInstall