opsmill/infrahub / .agents/skills/analyzing-ci-flakiness
analyzing-ci-flakiness
VerifiedAnalyzes recent CI failures on pull requests to identify flaky tests, using retry outcomes (failed attempt → green re-run) and cross-PR recurrence as evidence, and maintains a local longitudinal ledger so flakiness can be tracked over time. TRIGGER when: the user wants to find flaky tests, correlate recent CI failures, check which tests fail across PRs or recover on retry, or refresh the flakiness trend report. DO NOT TRIGGER when: babysitting a single PR's CI until green → monitoring-pull-requests; diagnosing or fixing one specific failing test → the bug-analysis skills.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 9.3 KB |
| scripts/collect.pyexecutable | script | 19.1 KB |
SKILL.md
View on GitHub---
name: analyzing-ci-flakiness
description: >-
Analyzes recent CI failures on pull requests to identify flaky tests, using retry outcomes
(failed attempt → green re-run) and cross-PR recurrence as evidence, and maintains a local
longitudinal ledger so flakiness can be tracked over time. TRIGGER when: the user wants to find
flaky tests, correlate recent CI failures, check which tests fail across PRs or recover on
retry, or refresh the flakiness trend report. DO NOT TRIGGER when: babysitting a single PR's CI
until green → monitoring-pull-requests; diagnosing or fixing one specific failing test → the
bug-analysis skills.
argument-hint: "Optional base-branch glob(s) and window, e.g. `release-1.11 14` (default: all bases, last 7 days)"
allowed-tools:
- Bash(python3 .agents/skills/analyzing-ci-flakiness/scripts/collect.py:*)
compatibility: Requires the gh CLI authenticated against the repo. Python 3 (stdlib only). Writes a cache under ~/ci-cache.
metadata:
version: 0.1.0
author: OpsMill
---
# CI Flakiness Analyzer
## Introduction
A test is *flaky* when its failure does not reproduce on the same code: the run was retried and
went green, or the same test fails on unrelated PRs. This skill mines both signals from GitHub
Actions history, downloads the failed job logs once into a local cache, and appends every
observation to a ledger (`~/ci-cache/<owner>-<name>/ledger.jsonl`) so repeated invocations —
weekly, or ad hoc — accumulate trend data instead of starting from scratch.
The mechanical part (fetching, caching, test-name extraction, known-signature classification) is
done by the bundled script. Your job is the judgment part: separating flakes from real
regressions, spotting new systemic signatures, and writing the report.
## Step 1 — Parse arguments
- Base-branch filter: any arguments that look like branch names or globs (`release-1.11`,
`release-*`, `stable`). Default: no filter (all PR bases), which is usually what "how flaky is
CI" means. Filter when the user names a branch.
- Window: a bare integer is a number of days (default 7). An ISO date means "since that date".
## Step 2 — Collect
Run the bundled collector (repo-root relative):
```bash
python3 .agents/skills/analyzing-ci-flakiness/scripts/collect.py \
[--base <glob> ...] [--days N | --since YYYY-MM-DD] [--repo owner/name]
```
It prints a JSON report to stdout and writes everything under
`~/ci-cache/<owner>-<name>/windows/<since>_<until>/`:
- `runs.jsonl` — every `pull_request` workflow run created in the window
- `failed_jobs_with_tests.json` — failed jobs of the interesting run-attempts, with extracted
failing tests, systemic-bucket tags, and a `recovered_same_run` flag
- `report-data.json` — headline numbers, ranked per-test table, per-bucket incident counts
(`bucket_incidents`: distinct jobs/runs/PRs per systemic bucket), and the ledger's weekly
history
- `joblogs/<job_id>.log` — raw logs (ANSI intact; strip with `sed 's/\x1b\[[0-9;]*m//g'`)
Notes the script already accounts for — don't re-derive them:
- The runs API's `pull_requests` field is empty for many runs; the script joins runs to PRs
through every PR head commit SHA as well. Don't trust the field alone.
- "Interesting attempts" = every earlier attempt of a retried run (that's what the retry fixed)
plus final attempts that failed. Runs cancelled on attempt 1 are concurrency noise and skipped.
- Logs already on disk are never re-downloaded; the ledger is deduplicated by (job, test). Old
logs expire on GitHub's side (~90 days) — an empty `joblogs/*.log` means expired, not passing.
## Step 3 — Investigate what the script could not name
For failed jobs with an empty `tests` list and no bucket tag, read the log yourself (grep for
`##[error]`, `FAILED`, `Error:`, `Timeout`). Two outcomes:
- It matches a *new* systemic signature (infra failure that cascades over many tests). Add a
regex for it to `BUCKETS` in `collect.py` and to the table below, so future runs classify it.
- It's a genuine test failure the extraction regexes missed — note the test manually and
consider extending `extract_tests`.
### Known systemic signatures (as of 2026-11 — keep in sync with `BUCKETS` in collect.py)
| Bucket | Signature | Meaning |
|---|---|---|
| `stack-readiness` | `ServerNotResponsiveError … /api/schema/load` | Seeded testcontainers stack not ready; the whole pytest-playwright shard errors. One incident, not N flaky tests. |
| `vitest-mock-corruption` | `TypeError: vi.mocked(...).mockX is not a function` | vitest browser-mode module-mocking race; hits a different test file each time. |
| `prefect-setup-triggers-timeout` | `Setup triggers` task `ReadTimeout` | Prefect hang at session setup; downstream tests hit their own timeouts. |
| `neo4j-deadlock` | `Neo.TransientError.Transaction.DeadlockDetected` | Concurrent-write deadlock, usually integration suites under xdist. |
| `compose-boot-failure` | `docker compose … up --wait` non-zero exit | Stack never booted; job-level infra failure. |
| `sqlite-locked` | `(sqlite3.OperationalError) database is locked` (also matches the raw `sqlite3.OperationalError:` form) | Prefect's sqlite under contention. |
| `runner-oom` | `Process completed with exit code 137` | Runner OOM/SIGKILL; the mass test failures in the same job are casualties, not flakes. |
| `docker-network-pool-exhausted` | `all predefined address pools have been fully subnetted` | Leaked compose networks exhausted the docker address pools on a self-hosted runner. |
| `actions-download-429` | `Failed to download action … 429` | GitHub rate-limited its own action download; pure platform flake. |
| `prefect-task-manager-wedged` | `RuntimeError: Prefect task manager setup already failed for http…` | The memoized task-manager setup (`backend/tests/helpers/task_manager.py`) timed out once against a Prefect test server; every later class fail-fasts on the remembered failure. One incident, hundreds of cascaded ERRORs. |
| `pytest-green-exit-1` | green pytest summary (no failed, no errors) directly followed by exit 1 | Session-teardown/plugin abort after all tests passed (e.g. testcontainers result reporting). |
| `pytest-session-timeout` | `xdist.dsession.Interrupted: session-timeout: 1800.0 sec exceeded` | A component shard ran past the pytest-xdist session budget; the summary shows only passes, the job exits 2, and the tests it did not reach are not reported. |
| `asyncio-stream-buffer-error` | `BufferError: Existing exports of data: object cannot be re-sized` (raised from `asyncio/streams.py` during the Neo4j Bolt handshake) | Seen in `backend-benchmark`: the shared `db` fixture fails to connect, so all 13 benchmarks error at setup. One incident per job, not 13 flaky tests. |
| `docker-pull-lease` | `unable to lease content: lease does not exist` within four lines of the step's `exit code 1` | The Docker daemon on a self-hosted runner lost a containerd lease while pulling an image (`demo.pull`); the job fails before any test runs. `version-upgrade.yml` retries the pull, and a pull that succeeded on a retry is not tagged. |
| `redis-read-timeout` | `redis.exceptions.TimeoutError: Timeout reading from …` in the server's output: on a Docker Compose line prefixed `infrahub-server-N \|`, or anywhere in the output of the `docker logs "${INFRAHUB_BUILD_NAME}-server-1"` step (`ci.yml`, `version-upgrade.yml`), which prints the server's log without that prefix, up to the next step | The server's cache stopped answering; the E2E fixtures that load data through the server error at setup. One incident, not N flaky tests. A unit test that raises the same message is not tagged. |
## Step 4 — Judge: flake vs regression
For each test in the ranked table, classify:
- **Flaky (strong)** — fails on ≥2 unrelated PRs, or `recovered_on_retry > 0`. The more distinct
PRs, the stronger.
- **Flaky (weak)** — single occurrence with an infra-flavored error (locator timeout, transient
branch not found) and the PR later went green. List, but rank low.
- **Suspect regression, not a flake** — the same test fails on *every* attempt of the same
commit and the PR is still red, or the failures started only after a specific merge. Say so
explicitly; do not bury it in the flake list. Cross-check: does the test fail on any PR that
does not contain the suspect change?
- **Systemic bucket** — tests whose only failures carry a bucket tag are casualties, not causes.
Report the bucket (with the incident count from `bucket_incidents` in `report-data.json`), not
the individual tests.
Different tests failing on successive attempts of the same run = two independent flakes, not a
regression.
## Step 5 — Report
Write `ANALYSIS.md` into the window directory, then give the user a summary. Lead with the
ranked flake candidates. Include:
1. Headline numbers: PRs in scope, runs matched, retried runs, retried-and-recovered runs
(pure-flake evidence), hard failures.
2. Ranked flake candidates — test id, distinct PRs/runs, recovered-on-retry count, one-line
error cause. Group systemic buckets as single entries.
3. Suspected real regressions, clearly separated.
4. Trend — from `weekly_history` in `report-data.json`: which offenders are new this window,
which recur week over week, which disappeared (likely fixed). This section is the reason the
ledger exists; don't skip it once ≥2 windows of data exist.
Do not propose fixes unless asked; the deliverable is the evidence-ranked candidate list.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |