Skip to content

opsmill/infrahub / .agents/skills/analyzing-ci-flakiness

analyzing-ci-flakiness

Verified

Analyzes recent CI failures on pull requests to identify flaky tests, using retry outcomes (failed attempt → green re-run) and cross-PR recurrence as evidence, and maintains a local longitudinal ledger so flakiness can be tracked over time. TRIGGER when: the user wants to find flaky tests, correlate recent CI failures, check which tests fail across PRs or recover on retry, or refresh the flakiness trend report. DO NOT TRIGGER when: babysitting a single PR's CI until green → monitoring-pull-requests; diagnosing or fixing one specific failing test → the bug-analysis skills.

CI Flakiness Analyzer

Introduction

A test is flaky when its failure does not reproduce on the same code: the run was retried and went green, or the same test fails on unrelated PRs. This skill mines both signals from GitHub Actions history, downloads the failed job logs once into a local cache, and appends every observation to a ledger (~/ci-cache/<owner>-<name>/ledger.jsonl) so repeated invocations — weekly, or ad hoc — accumulate trend data instead of starting from scratch.

The mechanical part (fetching, caching, test-name extraction, known-signature classification) is done by the bundled script. Your job is the judgment part: separating flakes from real regressions, spotting new systemic signatures, and writing the report.

Step 1 — Parse arguments

  • Base-branch filter: any arguments that look like branch names or globs (release-1.11, release-*, stable). Default: no filter (all PR bases), which is usually what "how flaky is CI" means. Filter when the user names a branch.
  • Window: a bare integer is a number of days (default 7). An ISO date means "since that date".

Step 2 — Collect

Run the bundled collector (repo-root relative):

python3 .agents/skills/analyzing-ci-flakiness/scripts/collect.py \
  [--base <glob> ...] [--days N | --since YYYY-MM-DD] [--repo owner/name]

It prints a JSON report to stdout and writes everything under ~/ci-cache/<owner>-<name>/windows/<since>_<until>/:

  • runs.jsonl — every pull_request workflow run created in the window
  • failed_jobs_with_tests.json — failed jobs of the interesting run-attempts, with extracted failing tests, systemic-bucket tags, and a recovered_same_run flag
  • report-data.json — headline numbers, ranked per-test table, per-bucket incident counts (bucket_incidents: distinct jobs/runs/PRs per systemic bucket), and the ledger's weekly history
  • joblogs/<job_id>.log — raw logs (ANSI intact; strip with sed 's/\x1b\[[0-9;]*m//g')

Notes the script already accounts for — don't re-derive them:

  • The runs API's pull_requests field is empty for many runs; the script joins runs to PRs through every PR head commit SHA as well. Don't trust the field alone.
  • "Interesting attempts" = every earlier attempt of a retried run (that's what the retry fixed) plus final attempts that failed. Runs cancelled on attempt 1 are concurrency noise and skipped.
  • Logs already on disk are never re-downloaded; the ledger is deduplicated by (job, test). Old logs expire on GitHub's side (~90 days) — an empty joblogs/*.log means expired, not passing.

Step 3 — Investigate what the script could not name

For failed jobs with an empty tests list and no bucket tag, read the log yourself (grep for ##[error], FAILED, Error:, Timeout). Two outcomes:

  • It matches a new systemic signature (infra failure that cascades over many tests). Add a regex for it to BUCKETS in collect.py and to the table below, so future runs classify it.
  • It's a genuine test failure the extraction regexes missed — note the test manually and consider extending extract_tests.

Known systemic signatures (as of 2026-11 — keep in sync with BUCKETS in collect.py)

BucketSignatureMeaning
stack-readinessServerNotResponsiveError … /api/schema/loadSeeded testcontainers stack not ready; the whole pytest-playwright shard errors. One incident, not N flaky tests.
vitest-mock-corruptionTypeError: vi.mocked(...).mockX is not a functionvitest browser-mode module-mocking race; hits a different test file each time.
prefect-setup-triggers-timeoutSetup triggers task ReadTimeoutPrefect hang at session setup; downstream tests hit their own timeouts.
neo4j-deadlockNeo.TransientError.Transaction.DeadlockDetectedConcurrent-write deadlock, usually integration suites under xdist.
compose-boot-failuredocker compose … up --wait non-zero exitStack never booted; job-level infra failure.
sqlite-locked(sqlite3.OperationalError) database is locked (also matches the raw sqlite3.OperationalError: form)Prefect's sqlite under contention.
runner-oomProcess completed with exit code 137Runner OOM/SIGKILL; the mass test failures in the same job are casualties, not flakes.
docker-network-pool-exhaustedall predefined address pools have been fully subnettedLeaked compose networks exhausted the docker address pools on a self-hosted runner.
actions-download-429Failed to download action … 429GitHub rate-limited its own action download; pure platform flake.
prefect-task-manager-wedgedRuntimeError: Prefect task manager setup already failed for http…The memoized task-manager setup (backend/tests/helpers/task_manager.py) timed out once against a Prefect test server; every later class fail-fasts on the remembered failure. One incident, hundreds of cascaded ERRORs.
pytest-green-exit-1green pytest summary (no failed, no errors) directly followed by exit 1Session-teardown/plugin abort after all tests passed (e.g. testcontainers result reporting).
pytest-session-timeoutxdist.dsession.Interrupted: session-timeout: 1800.0 sec exceededA component shard ran past the pytest-xdist session budget; the summary shows only passes, the job exits 2, and the tests it did not reach are not reported.
asyncio-stream-buffer-errorBufferError: Existing exports of data: object cannot be re-sized (raised from asyncio/streams.py during the Neo4j Bolt handshake)Seen in backend-benchmark: the shared db fixture fails to connect, so all 13 benchmarks error at setup. One incident per job, not 13 flaky tests.
docker-pull-leaseunable to lease content: lease does not exist within four lines of the step's exit code 1The Docker daemon on a self-hosted runner lost a containerd lease while pulling an image (demo.pull); the job fails before any test runs. version-upgrade.yml retries the pull, and a pull that succeeded on a retry is not tagged.
redis-read-timeoutredis.exceptions.TimeoutError: Timeout reading from … in the server's output: on a Docker Compose line prefixed infrahub-server-N |, or anywhere in the output of the docker logs "${INFRAHUB_BUILD_NAME}-server-1" step (ci.yml, version-upgrade.yml), which prints the server's log without that prefix, up to the next stepThe server's cache stopped answering; the E2E fixtures that load data through the server error at setup. One incident, not N flaky tests. A unit test that raises the same message is not tagged.

Step 4 — Judge: flake vs regression

For each test in the ranked table, classify:

  • Flaky (strong) — fails on ≥2 unrelated PRs, or recovered_on_retry > 0. The more distinct PRs, the stronger.
  • Flaky (weak) — single occurrence with an infra-flavored error (locator timeout, transient branch not found) and the PR later went green. List, but rank low.
  • Suspect regression, not a flake — the same test fails on every attempt of the same commit and the PR is still red, or the failures started only after a specific merge. Say so explicitly; do not bury it in the flake list. Cross-check: does the test fail on any PR that does not contain the suspect change?
  • Systemic bucket — tests whose only failures carry a bucket tag are casualties, not causes. Report the bucket (with the incident count from bucket_incidents in report-data.json), not the individual tests.

Different tests failing on successive attempts of the same run = two independent flakes, not a regression.

Step 5 — Report

Write ANALYSIS.md into the window directory, then give the user a summary. Lead with the ranked flake candidates. Include:

  1. Headline numbers: PRs in scope, runs matched, retried runs, retried-and-recovered runs (pure-flake evidence), hard failures.
  2. Ranked flake candidates — test id, distinct PRs/runs, recovered-on-retry count, one-line error cause. Group systemic buckets as single entries.
  3. Suspected real regressions, clearly separated.
  4. Trend — from weekly_history in report-data.json: which offenders are new this window, which recur week over week, which disappeared (likely fixed). This section is the reason the ledger exists; don't skip it once ≥2 windows of data exist.

Do not propose fixes unless asked; the deliverable is the evidence-ranked candidate list.

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall