Skip to content

zoltanmaric/coppersushi / .agents/skills/forecast-experiment

forecast-experiment

Verified

Records hypotheses, evidence and decisions when proposing, selecting, running or evaluating forecast experiments, including backtests and feature or model comparisons. Excludes routine tests and unchanged forecast operation.

Forecast experiments

Read the evaluation contract and point-in-time contract before designing a comparison.

Propose

Save each candidate as a brief page under wiki/price-forecasting/research/. Maintain index.md there with linked proposals, their status, relationships and the recommended next choice with its reason. Record in each proposal:

  • The hypothesis and intended change.
  • Motivating proposal links, MLflow run IDs and prior results inspected; use none for the first baseline.
  • Rough cost.

Keep proposals and selection decisions in the wiki. Create MLflow runs for computational executions, not for proposals or written reviews. If proposing only, save the proposal without launching work.

Select before execution

Read existing proposals and conclusions before recommending the next experiment. Compare relevant candidates by what they would teach, expected value, cost and prerequisites.

Recommend a candidate for the user to choose before execution, unless the user has explicitly delegated selection within an agreed research budget. Honor an existing choice or delegation without asking again.

Save the selection decision on the chosen proposal before execution. Link the candidates considered, explain the choice and deferrals, and record the user's choice or delegated budget. Update the research index; keep deferred proposals available.

Before execution, complete and freeze the selected proposal's evaluation plan in Git: comparison baseline, fixed evaluation dates and metrics, and what result would support the hypothesis.

During execution

At execution start, attach the frozen proposal, including its selection decision, to the MLflow run with its wiki path and Git revision. Copy it; keep the wiki original. Link folds, variants and retries to that proposal version; distinguish these executions from motivating runs.

Give runs descriptive human-readable names; retain stable IDs for links and storage. Link the exact run manifests for inputs, code, environment, configuration and seeds; do not copy their contents into prose. Retain failed and interrupted attempts with their status. Preserve each run's frozen proposal; use a revised proposal and new linked execution when the hypothesis or evaluation protocol changes.

After execution

Attach a conclusion to the execution run with results, evidence links, limitations and the next decision: keep, reject or investigate further. Compare against the baseline on matching evaluation rows. Record losses and failures as well as gains. If reviewing an existing run without a prior proposal, mark the rationale as retrospective; do not invent a pre-run hypothesis.

Update the wiki proposal with a short outcome and links to its executions and conclusions; refresh the research index. Revisit deferred proposals in light of the results; link new proposals to their motivating evidence.

Identify evaluation periods whose results informed experiment choices as development data, not untouched holdouts. Keep execution evidence in MLflow and research synthesis in the wiki, linked by run IDs. Follow this procedure manually when automation is absent.

Before closing an investigation, promote anything required by maintained code into explicitly owned configuration, data contracts or test fixtures; remove production dependencies on investigation folders. Retire scripts used only for the completed investigation after verifying that their contents exist in a commit reachable from main, and update reproduction links to that revision. If the scripts have not reached main, retain them through the first merge and delete them in a follow-up PR; adding and deleting within one squash-merged PR does not preserve them in main’s history.

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall