Skip to content

tobihagemann/turbo / codex/skills/investigate

investigate

Verified

Find the root cause of bugs, test failures, build errors, or performance issues by testing hypotheses. Use when the user asks to "investigate this bug", "debug this", or "find the root cause".

Investigate

Systematic methodology for finding the root cause of bugs, failures, and unexpected behavior. Cycle through characterize-isolate-hypothesize-test steps, with oracle escalation for hard problems. Diagnose the root cause — do not apply fixes.

Optional: $ARGUMENTS contains the problem description or error message.

Step 1: Characterize

Gather the symptom and establish what is actually happening:

  1. Collect evidence — error message, stack trace, test output, log entries, or user description of unexpected behavior
  2. Classify the problem type:
SignalType
Stack trace / exceptionRuntime error
Test assertion failureTest failure
Compilation / bundler / build errorBuild failure
Type checker error (tsc, mypy, pyright)Type error
Slow response / high CPU / memory growthPerformance
"It does X instead of Y" / no errorUnexpected behavior
  1. Establish reproduction — run the failing command, test, or operation, and confirm its output shows the reported symptom rather than a neighboring failure. When no command at hand shows the symptom, build one, trying whichever of these fit, in this order:
    • A test at the boundary that reaches the failure
    • A script that drives the running system from outside and asserts on what it returns
    • A captured request, payload, or event log replayed through the failing path
    • The same input run through a known-good version and the current one, with the outputs compared
    • Many generated inputs, when the failure appears for only some
    • For an intermittent failure, the trigger repeated or run concurrently until it fails often enough to test against

Reproduction is established when one command has been run and its output shows the reported symptom. Record that command and its output for verification. For intermittent or long-running reproductions, tail logs in a background shell, filtered for relevant signals (errors, stack traces, specific identifiers) so failures surface live while you work.

When neither the failing command nor any reproduction built from the list above shows the symptom, output what was tried and what each attempt showed. Then use request_user_input to offer three ways forward, and act on the answer. When request_user_input cannot reach the user, stop and report what was tried and that the way forward is unresolved.

  • Provide access or an artifact — the user supplies an environment where the failure occurs, or a capture from one such as a log, a trace, or a recorded request. Return to item 3 with it.
  • Allow temporary instrumentation — add labeled logging where the failure occurs, have the user trigger the failure, and read what it recorded. Remove the logging once it has been read, and confirm with git diff -- <file> that each file is back to its pre-check state. Carry what it recorded into Step 2 as evidence. When it yields a command whose output shows the symptom, record that command as the reproduction; otherwise continue as under Proceed without a reproduction.
  • Proceed without a reproduction — document the constraints and continue on historical evidence. Mark Step 4 verdicts and the report's root cause unverified by reproduction, and give the report's reproduction command as none, followed by what was tried. When a later step produces a command whose output shows the symptom, record that command from then on and drop the mark.

Minimize the Reproduction

Cut the reproduction down in a throwaway copy, leaving the working tree and its git index unchanged. Remove inputs, configuration, data, callers, and steps, halving what remains before cutting single elements. Re-run after each cut and restore a cut that makes the symptom disappear. Stop when removing any remaining element makes the symptom disappear, or once the reproduction runs quickly and its remaining elements can be named. Record the minimal command alongside the original, with what was cut, then delete the copy and report anything that could not be deleted. The original stays the report's reproduction command. Skip when the reproduction is a single test or command with nothing left to remove, when each run is slow or intermittent, or when no reproduction exists.

Step 2: Isolate

Narrow from "something is wrong" to "the problem is in this area." Read references/problem-type-playbooks.md for type-specific first moves and tool sequences.

Git Archeology

For all problem types, check what changed recently near the failure point:

git log --oneline -20 -- <file>
git blame -L <start>,<end> <file>

If a known-good state exists (e.g., "this worked yesterday"), consider git bisect to pinpoint the breaking commit.

When the problem description names a version, tag, or build that differs from the ref under investigation, resolve it to a ref and read what changed on the failing path between the two before generating hypotheses:

git diff <reported-ref>..HEAD -- <failing-path-files>

Read the full diff rather than its --stat summary. Carry each difference that could produce the symptom forward as a ranked hypothesis.

Upstream Issue Search

When the failure surfaces inside a third-party dependency, search its issue tracker for a distinctive string from the error before reading deeper into the dependency's code. An issue whose symptom matches often names the cause and the fix outright. Carry a match forward as a ranked hypothesis and test it.

Scope Narrowing

  • Stack traces: Read the throwing function and its callers — full functions, not just the flagged line
  • Test failures: Read both the test and the system under test
  • Build errors: Read the config file and the referenced source
  • Unexpected behavior: Trace the data flow from input to the unexpected output

Before treating a record, file, or build artifact as evidence of the system's behavior, confirm the system under test produced it: check creator, source metadata, or generation time. Suspect imported, seeded, hand-edited, and leftover data from an earlier run, which reads identically to generated output. A checkout of another repository is the same trap: confirm it is current before reading it as evidence, since a stale one reads identically to the authoritative source.

Step 3: Hypothesize

Before forming a hypothesis about the machinery around a failure, such as a toolchain version, a configuration policy, or an environment difference, read the failing line, identify every path, package, symbol, or resource it names, and confirm each one resolves. Error text often names the site that consumed a missing input rather than the input itself, so the surrounding machinery looks responsible when it is not. Rank a machinery hypothesis only after every named reference checks out.

Once every named reference resolves and the operation has never once succeeded, rank a refusal ahead of any race or resource-exhaustion hypothesis: a denied permission, a firewall rule, an allowlist, an expired or missing credential. Intermittent failure is what a race or a contended resource usually looks like, so a run of attempts with zero successes ranks both below a refusal. Carry the hard blocks on the failing path into Step 4 as the first hypotheses to test, ahead of any measurement or instrumentation.

Generate 2-4 hypotheses ranked by likelihood. Each hypothesis must be falsifiable — specify what evidence would confirm or refute it.

Format:

H1 (most likely): [description] — confirmed if [X], refuted if [Y]
H2: [description] — confirmed if [X], refuted if [Y]
H3: [description] — confirmed if [X], refuted if [Y]

Check that the observed case can discriminate: when confirming and refuting evidence would look identical in it, the case is degenerate and any verdict drawn from it is inconclusive. Degenerate cases hide the difference they are supposed to reveal, such as a scaling factor of 1, a single-element collection, or an identity transform. Find a non-degenerate case, or construct one as a Step 4 experiment.

Parallel Investigation

For complex problems with 3+ hypotheses and a non-obvious root cause, spawn parallel investigators simultaneously.

Spawn condition: 3+ hypotheses AND the problem is not a simple typo, missing import, or syntax error.

Skip when 1-2 hypotheses are obvious (e.g., stack trace points directly to the bug).

Before dispatching, read the project's test configuration and CI workflow to identify any test tier that resets a shared external resource between tests, such as a database, a fixed port, or a cache. Such tiers have no cross-process interlock, so branches running them concurrently wipe each other's state and return failures that look like real defects. Name any such tier to every branch as off-limits.

When the evidence lives in a repository other than this one, including a submodule or a vendored clone with its own remote, establish the authoritative ref before dispatching and bring it up to date, fetching it or reading it through the forge API. A local checkout may be behind its remote, and an investigation branch reading a stale one returns findings the current code has already resolved. Name that ref and how to read it in every branch prompt, including the text the Claude consultation branch forwards.

Launch all investigation branches with spawn_agent / wait_agent using inherited model defaults, issuing every call in one batch. Do not issue one and await its result before issuing the rest. Expect one branch per hypothesis plus one Claude consultation branch. Every branch prompt must direct it to treat the shared working tree and its git index as read-only and to gather evidence by reading and reasoning; experiments that mutate code wait for Step 4, where they run one at a time. HEAD stays where it is: read other refs with git show <ref>:<path> rather than git checkout or git switch.

  • Hypothesis branch (one per hypothesis): Each receives the hypothesis, relevant file paths, what evidence to look for, and instructions to report confirmed / refuted / inconclusive with evidence. Budget: max 5 tool calls per branch.
  • Claude consultation branch: Run the $consult-claude skill with a focused prompt describing the problem, reproduction, and files examined. The external perspective can dig into patterns the hypothesis-driven branches miss. Run the $evaluate-findings skill on its output after the consultation returns.

Once wait_agent has returned every hypothesis branch's report and only the Claude consultation branch has not reported, run the Step 4 actions that change nothing in the working tree, its git index, or anything the consultation's prompt points it at, and state each result as it lands. Then call wait_agent until the consultation's report arrives. The merge, every other Step 4 action, the Iteration decision, and the Investigation Report wait for that report.

After all investigators complete, merge results. Claude findings that overlap with a confirmed hypothesis reinforce confidence. Novel Claude findings become additional hypotheses to test in Step 4.

Step 4: Test

Verify each hypothesis with minimal, targeted actions:

Action TypeTool
Find usage or patternGrep
Read surrounding codeRead
Check recent changesBash (git log, git blame, git diff)
Run isolated testBash (specific test command)
Check dependency versionBash (npm ls, pip3 show, etc.)
Inspect runtime stateBash (add temporary logging, run, check output)
Vary one suspected variableBash (construct a throwaway fixture, run, compare)

When read-only evidence cannot discriminate, construct minimal throwaway fixtures that vary one suspected variable at a time. Exercise the system's inputs, and leave the working tree and its git index unchanged. Label each fixture clearly, delete them once the experiment concludes, and report anything that could not be deleted. When a check edits a tracked file instead, such as adding temporary logging, remove the edit once the check concludes and confirm with git diff -- <file> that the file is back to its pre-check state before recording the result. Write to an external or live system only after explicit user approval via request_user_input, stating the target system, every record the write will touch including those reached through triggers, cascades, and hooks, and the cleanup plan. When request_user_input does not reach the user, write nothing and report the approval as unresolved.

When a hypothesis names a timing window behind an intermittent failure, make the failing order deterministic in a throwaway copy of the test or operation, such as with a synchronous stall spanning that window, and confirm the copy fails with exactly the recorded symptom. Once it does, give as the report's reproduction command the command that runs the copy, with where the stall goes and how long it spans, so the copy can be rebuilt once deleted.

Record each result:

HypothesisVerdictEvidence
H1confirmed / refuted / inconclusive[what was found]
H2confirmed / refuted / inconclusive[what was found]

Iteration

If all hypotheses are refuted or inconclusive:

  1. Document what was learned — each refuted hypothesis eliminates a possibility and narrows the search
  2. Return to Step 2 with the new information to re-isolate
  3. Generate new hypotheses in Step 3 based on updated understanding

Cycle budget: maximum 2 full cycles (hypothesize → test → learn → repeat) before escalating.

Escalation

After 2 failed hypothesis cycles, offer escalation to $consult-oracle via request_user_input:

Investigation stalled after [N] hypothesis cycles.

Tested: [summary of hypotheses and evidence]
Remaining unknowns: [what is still unclear]

Escalate to Oracle? (consults external model with full context)

Proceed only if the user approves.

Investigation Report

Output results as text:

Investigation Report:

Problem: [one-line description]
Type: [runtime error | test failure | build failure | type error | performance | unexpected behavior]
Root cause: [confirmed cause, or "unresolved" with best hypothesis; append "unverified by reproduction" when the mark applies]

Evidence:
- [what confirmed the root cause]

Suggested fix: [description of what to change, or "needs further investigation"]
Reproduction command: [command to verify the fix once applied, or "none" followed by what was tried]

Hypotheses tested:
1. [hypothesis] — [confirmed/refuted/inconclusive, plus "unverified by reproduction" when the mark applies] — [evidence]
2. [hypothesis] — [confirmed/refuted/inconclusive, plus "unverified by reproduction" when the mark applies] — [evidence]

Escalation: [none | oracle]

Then call update_plan to mark this step completed and continue with the next step of the active workflow.

Rules

  • If the problem turns out to be environmental (wrong language runtime version, a declared dependency not installed locally, OS-specific), report that clearly — it may not require a code fix. A dependency the project never declared is a manifest defect, so report that as a code fix instead.
  • If the problem is in a dependency (not the project's code), document the dependency issue and suggest the options that leave the dependency's code unmodified: workarounds, and a report upstream unless an existing issue covers the defect or the dependency's latest release no longer has it.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall