K-Dense-AI/scientific-agent-skills / skills/scholar-evaluation
scholar-evaluation
VerifiedProvides qualitative-first, evidence-traceable developmental review of scholarly works and audit low-stakes research-assessment rubrics with optional local quality controls. Never use for ranking people or consequential decisions.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.1 KB |
| assets/evaluation_template.json | config | 1.3 KB |
| assets/evidence_manifest_template.json | config | 1.8 KB |
| assets/process_checklist_template.json | config | 2.1 KB |
| assets/ratings_template.csv | other | 1.7 KB |
| assets/rubric_template.json | config | 11.9 KB |
| references/evaluation_framework.md | markdown | 10.4 KB |
| references/local_tooling.md | markdown | 9.1 KB |
| references/responsible_assessment.md | markdown | 9.6 KB |
| references/security_validation.md | markdown | 4.2 KB |
| references/source_ledger.md | markdown | 13.8 KB |
| scripts/_common.py | script | 35.6 KB |
| scripts/calculate_scores.py | script | 1.8 KB |
| scripts/check_process.py | script | 8.1 KB |
| scripts/check_traceability.py | script | 8.9 KB |
| scripts/generate_report_scaffold.py | script | 9.4 KB |
| scripts/summarize_agreement.py | script | 11.1 KB |
| scripts/validate_rubric.py | script | 1.7 KB |
| scripts/weight_sensitivity.py | script | 10.0 KB |
references/security_validation.md
View on GitHub# Security Validation Record
Historical validation date: **2026-07-23**
The results below describe the earlier remediation, not a fresh scan. The
2026-10-01 refresh adds synthetic regression coverage for evaluation-round
agreement, bounded pair aggregation, scale consistency, unrounded sensitivity,
and companion membership. Collection-wide security scanning is recorded by
the refresh coordinator; do not infer a current scan from this historical record.
## Baseline
The repository `SECURITY.md` entry recorded **10 findings** with maximum
severity **CRITICAL**:
- four CRITICAL cross-file, environment, and network-exfiltration findings;
- three MEDIUM credential, prompt, and environment-harvesting findings; and
- three LOW cross-skill, command, and resource-use findings.
The affected files were the two former schematic-generation scripts and the
old `SKILL.md`.
## Remediation
- Deleted both schematic-generation scripts.
- Removed all network requests, API-key handling, environment access,
environment-file loading, third-party model behavior, image handling,
child-process execution, cross-skill invocation, and mandatory figure
instructions.
- Replaced the former recommendation-producing score calculator with bounded,
transparent descriptive rubric math.
- Added a strict prohibition on automated or assisted hiring, promotion,
tenure, admissions, funding, awards, discipline, person ranking, and other
high-impact personnel decisions.
- Added qualitative-first metric and prestige safeguards.
- Added construct, provenance, content-evidence, rater, agreement, uncertainty,
missingness, not-applicable, traceability, sensitivity, subgroup, conflict,
appeal, accessibility, privacy, and accountable-human controls.
- Added dependency-free, bounded local JSON/CSV tools with duplicate-key,
unknown-field, size, depth, non-finite-number, symbolic-link, and private
application-field rejection.
- Added minimized reports that never copy source-document content or emit rater
identifiers.
- Added static AST tests that prohibit network libraries, dynamic-code calls,
executable serialization, process launching, environment access, and the
deleted schematic files.
## Validation results
- Agent Skills reference validator: **PASS**
- Dependency-free CLI help checks: **8 passed**
- Synthetic standard-library tests: **29 passed**
- Explicit AST parse with bytecode disabled: **8 scripts parsed**
- Bytecode artifacts: **0**
- IDE lints: **0**
- Documented local-path check: **PASS**
- External source links: **24 passed**
- Direct behavioral security scan: **SAFE, 0 findings**
- Pull-request gate with `--fail-on HIGH`: **PASS**
- CRITICAL: 0
- HIGH: 0
- LOW: 2 (latest final run; LOW-only LLM wording varied between runs)
## Residual LOW findings
The latest LLM-assisted pull-request scan reported:
1. **Bash is broad but constrained.** Informational: the manifest declares Bash
because the documented fixed `python3` commands are shell invocations. The
body limits Bash to those local commands, and the direct behavioral scan
confirms no process-launching code in the scripts.
2. **Invented missing-file aliases.** False positive: the scanner claimed
inconsistent alternate directories for bundled assets and references. A
direct text search found no alternate template-directory path; every
documented local path is under the actual `assets`, `references`, or
`scripts` directory; and the deterministic path-resolution test resolves
every backticked local path.
The direct behavioral scan is clean. The residual findings neither permit data
transmission nor create a missing-file fallback. The generated root
`SECURITY.md` snapshot was intentionally not edited in this scoped refresh.
## Reproduction
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover \
-s tests/scholar-evaluation -p 'test_*.py' -v
for script in skills/scholar-evaluation/scripts/*.py; do
PYTHONDONTWRITEBYTECODE=1 python3 "$script" --help >/dev/null || exit 1
done
uv run skills-ref validate skills/scholar-evaluation
uv run skill-scanner scan skills/scholar-evaluation --use-behavioral
uv run python scan_pr_skills.py \
--fail-on HIGH \
--output /tmp/scholar-evaluation-pr-scan.md \
skills/scholar-evaluation
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |