joshua-zyy/academic-paper-writer / skills/academic-experiments
academic-experiments
VerifiedAudit, run, or verify experimental evidence for CS/AI/ML papers. Produces Evidence Inventory with evidence_type annotations (newly_run/preexisting_artifact/user_claim) and Protocol Risk assessments. Use when: checking if experiment results are reproducible, auditing existing experiment artifacts, running minimal reproducible commands, evaluating checkpoints without full retraining, documenting protocol risks like data leakage or missing baselines. Triggers on: 复核实验, run experiments, 实验结果, experiment evidence, verify results, 实验验证, evidence inventory, protocol risk, 跑实验, check results, reproduce experiments, 实验审计.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.1 KB |
| agents/experiment_agent.md | markdown | 5.9 KB |
| manifest.yaml | config | 1.2 KB |
| references/evidence-inventory.md | markdown | 1.7 KB |
| references/protocol-risks.md | markdown | 1.8 KB |
| references/run-strategy.md | markdown | 1.4 KB |
| scripts/evidence_scanner.py | script | 4.2 KB |
| static/core/anti-patterns.md | markdown | 571 B |
| static/core/output-contract.md | markdown | 906 B |
| static/core/red-lines.md | markdown | 477 B |
| static/core/stance.md | markdown | 2.5 KB |
references/evidence-inventory.md
View on GitHub# 证据盘点指南
## 证据优先级
按可靠性从高到低区分三类:
1. **`newly_run`** — 当前会话中执行并记录了命令与输出。正文中的定量结果优先来自此类。
2. **`preexisting_artifact`** — 仓库中已存在的 checkpoint、日志、CSV、图表、表格,经检查后确认来源和含义。若无法重跑,可使用此类但必须标注来源与限制。
3. **`user_claim`** — 仅来自用户口述或旧草稿叙述,尚未本地复核。只能写成待核验信息,不能当作最终结果。
## 盘点清单
在跑命令前,先检查以下资源是否存在:
| 类别 | 文件/目录 | 关注点 |
|------|----------|--------|
| 配置 | `config.*` | 超参数、数据路径、模型结构 |
| 入口 | `main.*` / `train.*` / `eval.*` | 训练/评估入口 |
| 脚本 | `run_*.sh` / `run_*.ps1` | 可复现的运行命令 |
| 环境 | `requirements.txt` / `environment.yml` | 依赖与环境 |
| 模型 | `checkpoints/` / `*.pth` / `*.ckpt` | 已有模型权重 |
| 输出 | `logs/` / `outputs/` / `results/` | 运行日志与结果 |
| 指标 | `*.csv` / `*.json` | 量化指标文件 |
| 图表 | `*.png` / `*.pdf` / `*.svg` | 可视化产物 |
## 盘点四个核心问题
完成盘点后,回答:
1. **能否只评估已有 checkpoint,而无需重训?**
- 优先路线:加载已有 checkpoint → 运行评估脚本 → 记录指标
2. **已有结果是窗口级、样本级还是受试者级?**
- 这决定了结论可推广的粒度
3. **当前 split 协议是否可靠?**
- 检查:训练/验证/测试划分方式、是否存在泄漏
4. **哪些结果可以在合理时间内复核?**
- 评估脚本通常几分钟;完整训练可能需要数小时或数天
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |