zrwangems/dsh-zrwang-plugin / ars/skills/sr-screener
sr-screener
Flagged系统综述文献筛选:把方案或 PROSPERO 协议转成资格规则,双盲筛选标题摘要与全文,第三人裁决冲突,输出 PRISMA 2020 计数、EndNote/Zotero RIS 分组与可交付给论文技能的 literature_corpus。Triggers: systematic review screening, screen papers, PRISMA counts, pilot screening, adjudicate conflicts | 文献筛选, 系统综述筛选, PRISMA 计数, 筛选冲突裁决。
scripts/check_benchmark_report.py
View on GitHub#!/usr/bin/env python3
"""Validate an ARS benchmark report against benchmark_report.schema.json.
Usage: python scripts/check_benchmark_report.py path/to/report.json
Exit 0 on pass (warnings may still be printed to stderr).
Exit 1 on validation failure.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
import jsonschema
SCHEMA_PATH = Path(__file__).resolve().parent.parent / "shared" / "benchmark_report.schema.json"
def load_schema() -> dict:
return json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))
def validate(report: dict) -> list[str]:
schema = load_schema()
validator = jsonschema.Draft202012Validator(schema)
return [f"{list(e.absolute_path)}: {e.message}" for e in validator.iter_errors(report)]
def warn_self_scored(report: dict) -> list[str]:
warnings = []
scoring = report.get("metrics", {}).get("scoring_independence")
if scoring == "self-scored":
warnings.append(
"WARNING: metrics.scoring_independence = 'self-scored'. "
"Self-scoring is permitted but a known reward-hacking risk. "
"Consider blind-scored or third-party-scored instead."
)
sample = report.get("human_baseline", {}).get("sample_size", 0)
if isinstance(sample, int) and 1 <= sample <= 2:
warnings.append(
f"WARNING: human_baseline.sample_size = {sample}. "
"n<=2 is dramatic, not statistical. Consider n>=5 or explicit disclosure in caveats."
)
return warnings
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("report", type=Path, help="Path to the benchmark report JSON")
args = parser.parse_args()
try:
report = json.loads(args.report.read_text(encoding="utf-8"))
except (FileNotFoundError, json.JSONDecodeError) as exc:
print(f"ERROR: failed to load {args.report}: {exc}")
return 1
errors = validate(report)
if errors:
for e in errors:
print(f"ERROR: {e}")
print(
f"\n{len(errors)} schema violation(s). "
f"See shared/benchmark_report_pattern.md for field rationale.",
file=sys.stderr,
)
return 1
for w in warn_self_scored(report):
print(w, file=sys.stderr)
print("OK: benchmark report validates.")
return 0
if __name__ == "__main__":
sys.exit(main())
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| add-product-e2e-evalpaperclipai/paperclipAdd or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| add-runner-evalpaperclipai/paperclipAdd or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| diagnose-why-work-stoppedpaperclipai/paperclipCompatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |
| fix-failed-paperclip-runpaperclipai/paperclipDiagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. | Stars: 115k | Updated | Claude CodeCodex | Verified | Install |