Skip to content

opsmill/infrahub / .agents/skills/analyzing-ci-flakiness

analyzing-ci-flakiness

Verified

Analyzes recent CI failures on pull requests to identify flaky tests, using retry outcomes (failed attempt → green re-run) and cross-PR recurrence as evidence, and maintains a local longitudinal ledger so flakiness can be tracked over time. TRIGGER when: the user wants to find flaky tests, correlate recent CI failures, check which tests fail across PRs or recover on retry, or refresh the flakiness trend report. DO NOT TRIGGER when: babysitting a single PR's CI until green → monitoring-pull-requests; diagnosing or fixing one specific failing test → the bug-analysis skills.

FileKindSize
SKILL.mdskill md9.3 KB
scripts/collect.pyexecutablescript19.1 KB

scripts/collect.py

View on GitHub
#!/usr/bin/env python3
"""Collect CI failure data for flakiness analysis, incrementally, into a local cache.

Fetches GitHub Actions runs for pull requests, matches them to PRs (via the
runs' ``pull_requests`` field *and* a head-SHA join, since the field is often
empty), identifies retried runs and failed attempts, downloads the failed job
logs, extracts failing test identifiers, classifies known systemic failure
signatures, and appends everything to a longitudinal ledger so successive
invocations build trend data.

Only stdlib + the ``gh`` CLI (must be authenticated). Safe to re-run: the runs
listing is refreshed each time, but job logs already on disk are never
re-downloaded and the ledger is deduplicated.

Usage:
  collect.py [--repo OWNER/NAME] [--base GLOB ...] [--days N | --since YYYY-MM-DD]
             [--cache DIR]

Outputs (under <cache>/<owner>-<name>/):
  ledger.jsonl                     one record per (job, test) ever observed
  windows/<since>_<until>/         this invocation's window
    runs.jsonl                     all pull_request runs created in the window
    failed_jobs_with_tests.json    failed jobs of interesting attempts + tests
    report-data.json               ranked frequency table, per-bucket incident counts + headline numbers
    joblogs/<job_id>.log           raw logs of failed jobs (ANSI intact; empty = expired)
"""

from __future__ import annotations

import argparse
import datetime as dt
import fnmatch
import json
import re
import subprocess  # noqa: S404
import sys
from collections import defaultdict
from pathlib import Path
from typing import TypedDict

# The raw job log keeps every terminal escape sequence, not only colour (SGR) codes. Strip general
# CSI sequences (colour, cursor moves, erase-line/screen) and OSC sequences (window title) so none
# survive into the test-extraction regexes.
ANSI = re.compile(
    r"\x1b\[[0-?]*[ -/]*[@-~]"  # CSI: ESC [ ... final-byte
    r"|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)"  # OSC: ESC ] ... (BEL | ST)
)

# The Actions list-runs API silently returns at most this many results per query.
API_RESULT_CAP = 1000

# Playwright's breadcrumb separator (U+203A) as it appears in job logs.
PW_SEP = "\u203a"

# Known systemic failure signatures. When one matches a job log, the job is
# tagged with the bucket. The tags feed the report's per-bucket incident counts
# (``bucket_incidents``) and the judgment step (SKILL.md Step 4), which reports
# a bucketed cascade as one incident rather than N flaky tests; the per-test
# table still lists every test, annotated with its buckets, so casualties can
# be discounted. Keep in sync with the table in SKILL.md.
BUCKETS: list[tuple[str, str]] = [
    ("stack-readiness", r"ServerNotResponsiveError: Unable to read from '[^']*/api/schema/load"),
    ("vitest-mock-corruption", r"TypeError: (?:vi\.mocked\(\.\.\.\)|\w+)\.mock\w+ is not a function"),
    ("prefect-setup-triggers-timeout", r"'Setup triggers'.*ReadTimeout|Task run encountered an exception ReadTimeout"),
    ("neo4j-deadlock", r"Neo\.TransientError\.Transaction\.DeadlockDetected"),
    ("compose-boot-failure", r"'docker', 'compose'.*'up', '--wait'.*non-zero exit status"),
    ("sqlite-locked", r"sqlite3\.OperationalError[):] database is locked"),
    ("runner-oom", r"Process completed with exit code 137|exit code: 137"),
    ("docker-network-pool-exhausted", r"all predefined address pools have been fully subnetted"),
    ("actions-download-429", r"Failed to download action .*429"),
    # One Prefect test server stopped answering, and every later test class fail-fasts on the
    # remembered failure. Matched through "RuntimeError: " so that a test asserting on this
    # message cannot tag a job by having its own source line printed in a traceback.
    (
        "prefect-task-manager-wedged",
        r"RuntimeError: Prefect task manager setup already failed for http",
    ),
    # pytest summary is green (no "N failed" and no "N errors") yet the process exits 1: a
    # session-teardown/plugin abort, e.g. the testcontainers result reporting. Errors have to be
    # excluded too, or a fixture cascade ("474 passed, 11 errors") reads as a green session.
    (
        "pytest-green-exit-1",
        r"=+ \d+ passed(?:(?!\d+ (?:failed|error))[^\n])*=+[^\n]*\n(?:[^\n]*\n){0,3}[^\n]*Process completed with exit code 1\.",
    ),
    # The shard ran out of its pytest-xdist session budget, so the job fails with every test it reached green.
    ("pytest-session-timeout", r"xdist\.dsession\.Interrupted: session-timeout: [\d.]+ sec exceeded"),
    # Raised in asyncio's stream reader during the benchmarks' Neo4j handshake, so every benchmark errors at setup.
    ("asyncio-stream-buffer-error", r"BufferError: Existing exports of data: object cannot be re-sized"),
    # Only a pull that ends the step counts, because a pull retried successfully leaves the same message in the log.
    (
        "docker-pull-lease",
        r"unable to lease content: lease does not exist[^\n]*\n(?:[^\n]*\n){0,3}[^\n]*##\[error\]Process completed with exit code 1\.",
    ),
    # Matched only in the server's output (Compose-prefixed, or inside the server-log step up to the next
    # step), because a unit test raises the same message on purpose.
    (
        "redis-read-timeout",
        r"infrahub-server-\d+\s+\|\s+redis\.exceptions\.TimeoutError: Timeout reading from"
        r'|##\[group\]Run docker logs "\$\{INFRAHUB_BUILD_NAME\}-server-1"[^\n]*\n'
        r"(?:(?!\S+ ##\[group\]Run )[^\n]*\n)*?[^\n]*redis\.exceptions\.TimeoutError: Timeout reading from",
    ),
]

LEDGER_FIELDS = (
    "run",
    "attempt",
    "final_conclusion",
    "recovered_same_run",
    "run_created",
    "workflow",
    "job_id",
    "job",
    "prs",
    "buckets",
)


def gh(args: list[str], *, check: bool = True) -> str:
    res = subprocess.run(["gh", *args], capture_output=True, text=True, errors="replace", check=False)  # noqa: S603, S607
    if res.returncode != 0 and check:
        raise RuntimeError(f"gh {' '.join(args[:3])}... failed: {res.stderr.strip()[:300]}")
    return res.stdout


def gh_json_lines(args: list[str]) -> list[dict]:
    out = gh(args)
    return [json.loads(line) for line in out.splitlines() if line.strip()]


def fetch_job_log(repo: str, job_id: int, log_path: Path) -> None:
    """Download one job log, distinguishing gone from transiently unavailable.

    On success the log is written to ``log_path``. On HTTP 404/410 (the log
    expired or was deleted on GitHub's side) an empty file is written as a
    durable sentinel so the job is never re-fetched. On any other failure
    (rate limit, network) — including a successful call with an empty body,
    which a real job log never has — nothing is written, so the next
    collection retries.
    """
    res = subprocess.run(  # noqa: S603
        ["gh", "api", f"repos/{repo}/actions/jobs/{job_id}/logs", "--allow-escape-sequences"],  # noqa: S607
        capture_output=True,
        text=True,
        errors="replace",
        check=False,
    )
    if res.returncode == 0 and res.stdout:
        log_path.write_text(res.stdout, encoding="utf-8")
    elif res.returncode == 0:
        print(f"[collect] WARN log {job_id}: empty response, leaving unfetched for retry", file=sys.stderr)
    elif "HTTP 404" in res.stderr or "HTTP 410" in res.stderr:
        log_path.write_text("", encoding="utf-8")
    else:
        print(f"[collect] WARN log {job_id}: {res.stderr.strip()[:300]}", file=sys.stderr)


def list_prs(repo: str, since: dt.date, base_globs: list[str]) -> list[dict]:
    # Look back further than the run window: a re-run in the window can belong
    # to a PR whose updatedAt predates it.
    pr_since = since - dt.timedelta(days=21)
    prs = json.loads(
        gh(
            [
                "pr",
                "list",
                "--repo",
                repo,
                "--state",
                "all",
                "--limit",
                "500",
                "--search",
                f"updated:>={pr_since.isoformat()}",
                "--json",
                "number,title,state,baseRefName,headRefName,updatedAt",
            ]
        )
    )
    if base_globs:
        prs = [p for p in prs if any(fnmatch.fnmatch(p["baseRefName"], g) for g in base_globs)]
    return prs


def pr_head_shas(repo: str, numbers: list[int]) -> dict[str, set[int]]:
    sha2pr: dict[str, set[int]] = defaultdict(set)
    for n in numbers:
        out = gh(["api", f"repos/{repo}/pulls/{n}/commits?per_page=100", "--paginate", "--jq", ".[].sha"], check=False)
        for sha in out.split():
            sha2pr[sha].add(n)
    return sha2pr


def _runs_query(repo: str, created: str) -> str:
    return f"repos/{repo}/actions/runs?event=pull_request&created={created}&per_page=100"


def list_runs(repo: str, since: dt.date, until: dt.date) -> list[dict]:
    """List runs in [since, until], splitting the date range to stay under the API's 1000-result cap."""
    jq = (
        ".workflow_runs[] | {id, name, head_branch, head_sha, run_attempt, "
        "conclusion, status, created_at, prs: [.pull_requests[] | "
        "{number, base: .base.ref}]}"
    )
    created = f"{since.isoformat()}..{until.isoformat()}"
    total = int(gh(["api", _runs_query(repo, created).replace("per_page=100", "per_page=1"), "--jq", ".total_count"]))
    if total > API_RESULT_CAP and since < until:
        mid = since + (until - since) // 2
        print(f"[collect] {total} runs in {created} exceeds the API result cap; splitting", file=sys.stderr)
        return list_runs(repo, since, mid) + list_runs(repo, mid + dt.timedelta(days=1), until)
    if total > API_RESULT_CAP:
        print(f"[collect] WARN {total} runs on {since} alone; the API returns only the newest results", file=sys.stderr)
    return gh_json_lines(["api", _runs_query(repo, created), "--paginate", "--jq", jq])


def failed_jobs_for_attempt(repo: str, run_id: int, attempt: int) -> list[dict]:
    return gh_json_lines(
        [
            "api",
            f"repos/{repo}/actions/runs/{run_id}/attempts/{attempt}/jobs?per_page=100",
            "--paginate",
            "--jq",
            '.jobs[] | select(.conclusion=="failure") | {id, name}',
        ]
    )


def extract_tests(job_name: str, text: str) -> list[str]:
    """Pull failing test identifiers out of a cleaned (ANSI-stripped) job log."""
    fails: set[str] = set()
    # pytest — backend suites and the pytest-playwright e2e suite
    fails.update(
        m.group(1).split(" - ")[0].rstrip(",")
        for m in re.finditer(r"(?:FAILED|ERROR) ((?:backend/)?tests/\S+::\S+)", text)
    )
    # legacy TS Playwright — numbered entries of the failure report
    if "E2E-testing-playwright" in job_name:
        for m in re.finditer(rf"\d+\)\s+\[[\w-]+\]\s+{PW_SEP}\s+(tests/e2e/[^\n{PW_SEP}]+){PW_SEP}([^\n]+)", text):
            spec = m.group(1).strip().split(":")[0]
            title = re.sub(r"\s+", " ", m.group(2)).strip()[:120]
            fails.add(f"PW {spec} {PW_SEP} {title}")
    # vitest browser mode
    if job_name == "frontend-tests":
        fails.update(
            f"VITEST {m.group(1)}" for m in re.finditer(r"FAIL\s+\|?\s*\w*\s*\|?\s+(src/\S+\.test\.\w+)", text)
        )
    return sorted(fails)


def classify(text: str) -> list[str]:
    return [name for name, pat in BUCKETS if re.search(pat, text)]


def match_runs_to_prs(runs: list[dict], pr_by_num: dict[int, dict], sha2pr: dict[str, set[int]]) -> list[dict]:
    matched = []
    for r in runs:
        nums = {p["number"] for p in r["prs"] if p["number"] in pr_by_num}
        nums |= {n for n in sha2pr.get(r["head_sha"], set()) if n in pr_by_num}
        if nums:
            r["pr_nums"] = sorted(nums)
            matched.append(r)
    return matched


def collect_failed_jobs(
    repo: str, targets: list[tuple[dict, int]], pr_by_num: dict[int, dict], win_dir: Path
) -> list[dict]:
    """Fetch failed jobs and their logs for each (run, attempt); build job entries."""
    jobs_out = []
    for r, attempt in targets:
        try:
            jobs = failed_jobs_for_attempt(repo, r["id"], attempt)
        except RuntimeError as exc:
            print(f"[collect] WARN jobs {r['id']}/{attempt}: {exc}", file=sys.stderr)
            continue
        for job in jobs:
            log_path = win_dir / "joblogs" / f"{job['id']}.log"
            if not log_path.exists():
                fetch_job_log(repo, job["id"], log_path)
            text = ANSI.sub("", log_path.read_text(errors="replace")) if log_path.exists() else ""
            jobs_out.append(
                {
                    "run": r["id"],
                    "attempt": attempt,
                    "final_attempt": r["run_attempt"],
                    "final_conclusion": r["conclusion"],
                    "recovered_same_run": attempt < r["run_attempt"] and r["conclusion"] == "success",
                    "run_created": r["created_at"],
                    "workflow": r["name"],
                    "prs": [{"number": n, "base": pr_by_num[n]["baseRefName"]} for n in r["pr_nums"]],
                    "job_id": job["id"],
                    "job": job["name"],
                    "tests": extract_tests(job["name"], text),
                    "buckets": classify(text),
                    "log_ok": bool(text.strip()),
                }
            )
    return jobs_out


def append_ledger(ledger_path: Path, jobs_out: list[dict], repo: str, today: dt.date) -> int:
    seen: set[str] = set()
    if ledger_path.exists():
        seen = {json.loads(line)["dedup_key"] for line in ledger_path.open(encoding="utf-8")}
    added = 0
    with ledger_path.open("a", encoding="utf-8") as fh:
        for entry in jobs_out:
            week = dt.datetime.fromisoformat(entry["run_created"]).strftime("%G-W%V")
            for test in entry["tests"] or [""]:
                key = f"{entry['job_id']}:{test}"
                if key in seen:
                    continue
                seen.add(key)
                record = {
                    "dedup_key": key,
                    "fetched_at": today.isoformat(),
                    "week": week,
                    "repo": repo,
                    "test": test,
                    **{k: entry[k] for k in LEDGER_FIELDS},
                }
                fh.write(json.dumps(record) + "\n")
                added += 1
    return added


class RankedTest(TypedDict):
    test: str
    distinct_runs: int
    distinct_prs: int
    attempts: int
    recovered_on_retry: int
    buckets: list[str]
    prs: list[int]


def ranked_tests(jobs_out: list[dict]) -> list[RankedTest]:
    freq: dict[str, list[dict]] = defaultdict(list)
    for e in jobs_out:
        for t in e["tests"]:
            freq[t].append(e)
    table: list[RankedTest] = [
        {
            "test": test,
            "distinct_runs": len({e["run"] for e in entries}),
            "distinct_prs": len({p["number"] for e in entries for p in e["prs"]}),
            "attempts": len(entries),
            "recovered_on_retry": sum(e["recovered_same_run"] for e in entries),
            "buckets": sorted({b for e in entries for b in e["buckets"]}),
            "prs": sorted({p["number"] for e in entries for p in e["prs"]}),
        }
        for test, entries in freq.items()
    ]
    table.sort(key=lambda x: (-x["distinct_prs"], -x["distinct_runs"], x["test"]))
    return table


def bucket_incidents(jobs_out: list[dict]) -> dict[str, dict[str, int]]:
    """Count distinct jobs/runs/PRs per systemic bucket, so a cascade reads as one incident."""
    jobs: dict[str, set[int]] = defaultdict(set)
    runs: dict[str, set[int]] = defaultdict(set)
    prs: dict[str, set[int]] = defaultdict(set)
    for e in jobs_out:
        for b in e["buckets"]:
            jobs[b].add(e["job_id"])
            runs[b].add(e["run"])
            prs[b].update(p["number"] for p in e["prs"])
    return {b: {"jobs": len(jobs[b]), "runs": len(runs[b]), "prs": len(prs[b])} for b in sorted(jobs)}


def weekly_history(ledger_path: Path) -> dict[str, dict[str, int]]:
    hist: dict[str, dict[str, int]] = {}
    if ledger_path.exists():
        for line in ledger_path.open(encoding="utf-8"):
            rec = json.loads(line)
            if rec["test"]:
                weeks = hist.setdefault(rec["test"], {})
                weeks[rec["week"]] = weeks.get(rec["week"], 0) + 1
    return {t: dict(sorted(w.items())) for t, w in sorted(hist.items())}


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--repo", default="opsmill/infrahub")
    ap.add_argument(
        "--base",
        action="append",
        default=[],
        help="base-branch glob(s) to keep, e.g. release-1.11 or 'release-*' (default: all)",
    )
    ap.add_argument("--days", type=int, default=7)
    ap.add_argument("--since", type=dt.date.fromisoformat)
    ap.add_argument("--cache", type=Path, default=Path.home() / "ci-cache")
    args = ap.parse_args()

    today = dt.datetime.now(tz=dt.UTC).date()
    since = args.since or today - dt.timedelta(days=args.days)
    repo_dir = args.cache / args.repo.replace("/", "-")
    win_dir = repo_dir / "windows" / f"{since.isoformat()}_{today.isoformat()}"
    (win_dir / "joblogs").mkdir(parents=True, exist_ok=True)

    pr_by_num = {p["number"]: p for p in list_prs(args.repo, since, args.base)}
    print(f"[collect] {len(pr_by_num)} PRs in scope (bases: {args.base or 'all'})", file=sys.stderr)

    runs = list_runs(args.repo, since, today)
    (win_dir / "runs.jsonl").write_text("".join(json.dumps(r) + "\n" for r in runs))
    print(f"[collect] {len(runs)} pull_request runs since {since}", file=sys.stderr)

    matched = match_runs_to_prs(runs, pr_by_num, pr_head_shas(args.repo, list(pr_by_num)))

    # Attempts worth reading: every earlier attempt of a retried run (those
    # failures are what the retry "fixed"), plus the final attempt when it
    # failed outright. Runs cancelled on attempt 1 are concurrency noise.
    targets = [(r, a) for r in matched for a in range(1, r["run_attempt"] + (r["conclusion"] == "failure"))]
    print(f"[collect] {len(matched)} runs matched to PRs, {len(targets)} run-attempts to inspect", file=sys.stderr)

    jobs_out = collect_failed_jobs(args.repo, targets, pr_by_num, win_dir)
    (win_dir / "failed_jobs_with_tests.json").write_text(json.dumps(jobs_out, indent=1))
    new_records = append_ledger(repo_dir / "ledger.jsonl", jobs_out, args.repo, today)

    report = {
        "window": {"since": since.isoformat(), "until": today.isoformat()},
        "base_filter": args.base or "all",
        "prs_in_scope": len(pr_by_num),
        "runs_matched": len(matched),
        "runs_retried": sum(1 for r in matched if r["run_attempt"] > 1),
        "runs_recovered_on_retry": sum(1 for r in matched if r["run_attempt"] > 1 and r["conclusion"] == "success"),
        "runs_failed_final": sum(1 for r in matched if r["conclusion"] == "failure"),
        "failed_jobs": len(jobs_out),
        "ranked_tests": ranked_tests(jobs_out),
        "bucket_incidents": bucket_incidents(jobs_out),
        "weekly_history": weekly_history(repo_dir / "ledger.jsonl"),
        "new_ledger_records": new_records,
    }
    (win_dir / "report-data.json").write_text(json.dumps(report, indent=1))
    print(json.dumps(report, indent=1))
    print(f"[collect] window dir: {win_dir}", file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall