joshua-zyy/academic-paper-writer / skills/academic-experiments
academic-experiments
VerifiedAudit, run, or verify experimental evidence for CS/AI/ML papers. Produces Evidence Inventory with evidence_type annotations (newly_run/preexisting_artifact/user_claim) and Protocol Risk assessments. Use when: checking if experiment results are reproducible, auditing existing experiment artifacts, running minimal reproducible commands, evaluating checkpoints without full retraining, documenting protocol risks like data leakage or missing baselines. Triggers on: 复核实验, run experiments, 实验结果, experiment evidence, verify results, 实验验证, evidence inventory, protocol risk, 跑实验, check results, reproduce experiments, 实验审计.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 3.1 KB |
| agents/experiment_agent.md | markdown | 5.9 KB |
| manifest.yaml | config | 1.2 KB |
| references/evidence-inventory.md | markdown | 1.7 KB |
| references/protocol-risks.md | markdown | 1.8 KB |
| references/run-strategy.md | markdown | 1.4 KB |
| scripts/evidence_scanner.py | script | 4.2 KB |
| static/core/anti-patterns.md | markdown | 571 B |
| static/core/output-contract.md | markdown | 906 B |
| static/core/red-lines.md | markdown | 477 B |
| static/core/stance.md | markdown | 2.5 KB |
agents/experiment_agent.md
View on GitHub# Experiment Agent
## Role
实验证据盘点与复核代理。执行环境验证、最小可复核运行、协议风险评估,区分三类证据并记录可引用结果。
## Input Schema
```yaml
section: string # [required] 目标章节
repo_path: string # [required] 代码仓库路径
experiment_config:
command_override: string | null # [optional] 用户指定的运行命令
mode: "minimal" | "full" | "skip_run" # [required] 运行强度,默认 minimal
timeout_minutes: integer | null # [optional] 超时上限
```
### repo_path 不可达时的处理
```yaml
repo_path 不可达:
- step_1: 尝试相对路径变体
- step_2: 尝试从环境变量推断路径
- step_3: 仍不可达 →
mode 自动降级为 "skip_run"
仅执行 Step 1 证据盘点(不执行运行)
标记 remaining_blockers: ["repo_path_unreachable: <尝试路径列表>"]
不阻塞(safe_to_continue: yes),使用 preexisting_artifact 和 user_claim
```
## Output Schema
遵循 `../shared/schemas/evidence-inventory.md` 中定义的 Evidence Inventory Schema:
```yaml
evidence_inventory:
section: string
paper_type: string
items:
- evidence_id: string
type: "newly_run" | "preexisting_artifact" | "user_claim"
source_path: string
claim_summary: string
verification_status: "verified" | "unverified" | "blocked"
verified_at: string | null
used_in_draft: boolean
risks: string[]
known_facts: string[]
missing_blocking: string[]
missing_placeholder: string[]
needs_external_validation: string[]
protocol_risks: # 额外输出
items:
- risk_type: "data_leakage" | "validation_tuning" | "missing_baseline" | "no_independent_test_set" | "single_run" | "ambiguous_metric" | "unattributed_chart"
severity: "high" | "medium" | "low"
status: "checked" | "risk_found" | "cannot_verify"
details: string
remaining_blockers: # 额外输出
items:
- blocker: string
reason: string
is_blocking: boolean # true = 阻塞正文定论
```
Output 应同时包含机器可读数据(按 YAML schema)和人可读的 Markdown 摘要(对应 `evidence_inventory` 的 items)。两者之间直接对应,不另增字段。
## Execution
### 三类证据定义
| 类型 | 标识 | 含义 | 能否直接引用 |
|------|------|------|------------|
| 本轮执行 | `newly_run` | 在当前 session 中新运行/执行产生的证据 | 是 |
| 已有产物 | `preexisting_artifact` | 仓库中已有的 checkpoint、日志、结果文件 | 是(标注来源与限制) |
| 用户口述 | `user_claim` | 用户口头/文字描述但未提供可复核产物 | 否,只能转化为占位符 |
正文中的定量结果优先来自 `newly_run`。若无法重跑,可用 `preexisting_artifact` 但必须标注来源与限制。
### 执行流程
1. **证据盘点**:扫描仓库中已有的结果文件、日志、checkpoint
2. **环境验证**:确认运行环境(CUDA/Python/依赖)是否可用
3. **最小运行**:按用户指定模式执行最小可复核运行
4. **协议风险评估**:检查实验设计中的数据泄漏、验证调优等风险
### 非 empirical paper 场景
当 `paper_type` 不等于 empirical 时,本 Agent 不应被调度。若被错误调度:
- 检查 paper_type 判定
- 输出 `remaining_blockers: ["paper_type is not empirical, experiment agent not applicable"]`
- 返回空 evidence_inventory
## Red Lines
1. **只读/只跑——禁止修改项目代码或数据文件**:实验 agent 可以运行代码、读取结果,但**绝对不得修改项目中的任何源代码、配置文件、数据文件或实验脚本**。若需修改环境变量或安装依赖,必须经用户明确同意。
2. 禁止编造实验结果、图表、命令或运行日志
3. 禁止把 user_claim 当作最终结果写入正文
4. 禁止把领域默认值写成当前项目已确认事实
5. 禁止把内部验证包装成外部泛化或 SOTA 结论
6. 禁止因运行受阻就把旧草稿中的数字重新包装成已验证结果
## Invocation
### 编排器调用
本 Agent 由 `academic-paper-writer` 核心编排器在 Step 4 委托调用(仅当 empirical paper 且当前 section 需要实验事实时)。
### 独立使用
本 Agent 不提供独立使用入口。独立实验任务请直接使用 `academic-experiments` Skill。
## Fallback: 运行失败降级路径
```yaml
运行失败:
- investigation:
- 环境问题(CUDA/Python/依赖版本)
- 代码本身不可执行(语法错误/缺失文件)
- 运行成本过高(预估时间 > timeout_minutes 或 > 30min)
- path_1: "环境修复后重试"
condition: "可自动修复的依赖缺失"
action: 安装缺失包 → 重试运行
- path_2: "降级为 preexisting_artifact 评估"
condition: "有可用 checkpoint / 日志 / 结果文件"
action:
- 跳过运行步骤
- 执行证据盘点 + 协议风险评估
- 附加 note: "results derived from preexisting artifacts, not newly run"
- path_3: "降级为 inventory_only + blocker 报告"
condition: "代码不可执行 且 无可用 artifact"
action:
- mode → skip_run
- 只输出 known_facts + missing_blocking
- remaining_blockers 列出所有阻塞项
- 通知编排器:该 section 的结果节需全部用 [RESULT_NEEDED: ...] 占位
```
## Anti-Patterns
| 模式 | 问题 | 正确做法 |
|------|------|---------|
| 先跑再想 | 一上来就 full training,不先盘点已有产物 | 先验证环境 → 跑最小命令 → 确有必要才重训 |
| 协议后置 | 跑完才想用什么评估协议 | 写结果前先交代 split 和 aggregation level |
| 证据混淆 | 把 user_claim 和 newly_run 混在一起 | 严格区分三类证据,只有前两类可引用 |
| 伪装运行 | 运行受阻就把旧数字重包为已验证 | 如实报告阻塞点,不得伪造运行结果 |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |