Skip to content

joshua-zyy/academic-paper-writer / skills/academic-experiments

academic-experiments

Verified

Audit, run, or verify experimental evidence for CS/AI/ML papers. Produces Evidence Inventory with evidence_type annotations (newly_run/preexisting_artifact/user_claim) and Protocol Risk assessments. Use when: checking if experiment results are reproducible, auditing existing experiment artifacts, running minimal reproducible commands, evaluating checkpoints without full retraining, documenting protocol risks like data leakage or missing baselines. Triggers on: 复核实验, run experiments, 实验结果, experiment evidence, verify results, 实验验证, evidence inventory, protocol risk, 跑实验, check results, reproduce experiments, 实验审计.

references/protocol-risks.md

View on GitHub
# 协议风险与失败处理

## 协议风险清单

主动记录以下风险(不得省略):

| 风险类型 | 说明 | 典型信号 |
|---------|------|---------|
| 数据泄漏 | 训练/测试划分不规范 | subject 出现在两个集合中 |
| 验证集调参 | 用验证集选阈值后报告测试性能 | 只有一个 split 且没有独立测试集 |
| baseline 缺失 | 缺少关键对比方法 | 只有同类深度方法,没有传统基线 |
| 无独立测试集 | 只用交叉验证或内部划分 | 没有 held-out test set |
| 单次运行 | 没有方差估计 | 只有一次运行的结果 |
| 指标定义模糊 | 未核对指标计算方式 | 自定义指标与标准定义不一致 |
| 图表溯源不明 | 不知道图表的数据来源 | 只有图但无对应脚本和 checkpoint |

## 写结果时的约束

写 Results / Experiments 时:

- 先写协议,再写指标
- 先交代 split 和 aggregation level,再给数字
- 如果结果来自旧 artifact,要写清"来自现有 checkpoint / 现有输出文件"
- 如果结果只支持内部验证,就写成 internal validation,不要升级成 generalization claim
- 如果发现协议有缺陷,应主动把这件事写入 Discussion / Limitations

## 失败处理

若代码无法运行:

1. **报告具体阻塞点** — 明确指出哪个步骤失败(环境缺失/数据缺失/依赖缺失/超长运行成本)
2. **记录已尝试命令** — 完整记录尝试过的命令和返回的错误
3. **标出缺失条件** — 需要的 Python 版本、CUDA 版本、未安装的包、缺失的数据文件
4. **降级处理** — 将结果节降级为"已知证据 + 待复核项"
5. **不等同于已验证** — 不要因为运行受阻,就把旧草稿中的数字重新包装成已验证结果
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall