K-Dense-AI/scientific-agent-skills / skills/scholar-evaluation
scholar-evaluation
VerifiedProvides qualitative-first, evidence-traceable developmental review of scholarly works and audit low-stakes research-assessment rubrics with optional local quality controls. Never use for ranking people or consequential decisions.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 12.1 KB |
| assets/evaluation_template.json | config | 1.3 KB |
| assets/evidence_manifest_template.json | config | 1.8 KB |
| assets/process_checklist_template.json | config | 2.1 KB |
| assets/ratings_template.csv | other | 1.7 KB |
| assets/rubric_template.json | config | 11.9 KB |
| references/evaluation_framework.md | markdown | 10.4 KB |
| references/local_tooling.md | markdown | 9.1 KB |
| references/responsible_assessment.md | markdown | 9.6 KB |
| references/security_validation.md | markdown | 4.2 KB |
| references/source_ledger.md | markdown | 13.8 KB |
| scripts/_common.py | script | 35.6 KB |
| scripts/calculate_scores.py | script | 1.8 KB |
| scripts/check_process.py | script | 8.1 KB |
| scripts/check_traceability.py | script | 8.9 KB |
| scripts/generate_report_scaffold.py | script | 9.4 KB |
| scripts/summarize_agreement.py | script | 11.1 KB |
| scripts/validate_rubric.py | script | 1.7 KB |
| scripts/weight_sensitivity.py | script | 10.0 KB |
references/source_ledger.md
View on GitHub# Dated Source Ledger
Refreshed on **2026-10-01** against the current official pages, source PDFs,
and ScholarEval project record below. This updates the 2026-07-23 review.
Policy-source verification and local synthetic-tool tests are distinct: no
researcher profiles, bibliometric datasets, authenticated services, external
model pipeline, or private work were evaluated.
The bundled workflow is local JSON/CSV only. It contains no scholarly-database,
author-identity, citation-count, authentication, pagination, or rate-limit API
contract. The shared collection citation's arXiv request is a metadata lookup,
not an evaluation service. Do not graft provider endpoints into this rubric.
## Responsible research assessment
### San Francisco Declaration on Research Assessment (DORA)
- **Primary source:** [Read the Declaration](https://sfdora.org/read/)
- **Origin:** developed in 2012; official page rechecked 2026-10-01.
- **Verified points:** assess work on its own merits; do not use journal-based
measures as surrogates for an article or a person's contribution; state
criteria explicitly; consider data, software, and other outputs; use
qualitative evidence; make metric methods transparent and account for field
and output-type variation.
- **Use here:** categorical ban on scored prestige proxies and a requirement
for explicit criteria, diverse evidence, and traceability.
### DORA quantitative-indicator guidance
- **Primary source:** [Guidance on the responsible use of quantitative
indicators](https://sfdora.org/resource/guidance-on-the-responsible-use-of-quantitative-indicators-in-research-assessment/)
- **Document:** [official PDF](https://sfdora.org/wp-content/uploads/2024/05/DORA_indicators_guidance.pdf)
- **Release:** 2024; official resource page accessible in 2026.
- **Verified points:** no indicator captures research quality in one number;
uses should be clear, transparent, specific, contextual, and fair. The
guidance addresses journal measures, citation counts, the h-index,
field-normalized indicators, and altmetrics; it warns about reductive,
aggregate, composite, lagging, field, career-stage, and bias effects.
- **Use here:** quantitative indicators are excluded from rubric scores. If
mentioned descriptively outside the tools, their purpose, data, coverage,
time window, field normalization, uncertainty, bias, and non-quality meaning
must be explicit.
### Leiden Manifesto
- **Primary source:** Hicks, Wouters, Waltman, de Rijcke, and Rafols,
[“Bibliometrics: The Leiden Manifesto for research
metrics”](https://doi.org/10.1038/520429a), *Nature* 520, 429–431.
- **Published:** 2015-04-22. Publisher metadata remains current. The publisher
full text was access-limited during this refresh; principles were checked
against the [author-linked Spanish translation](https://www.leidenmanifesto.org/uploads/4/1/6/0/41603901/manifiesto_cast.pdf)
hosted by the official manifesto site.
- **Verified points:** quantitative evaluation should support qualitative
expert assessment; measure against missions; protect locally relevant
research; account for field variation; keep data and analysis open and
verifiable; allow those evaluated to verify data; account for career age and database differences;
avoid false precision; recognize gaming and system effects; review indicators
regularly.
- **Use here:** contextualization, inspectability, uncertainty, fairness review,
and periodic revision.
### Agreement on Reforming Research Assessment / CoARA
- **Primary record:** [Agreement on Reforming Research
Assessment](https://doi.org/10.5281/zenodo.13480728), version 1.
- **Published:** 2022-07-20; Zenodo record modified 2024-08-29.
- **Current official overview:** [CoARA Agreement](https://www.coara.org/agreement/the-agreement-full-text/)
- **Verified points:** recognize diverse outputs, practices, activities, roles,
and careers; base assessment primarily on qualitative judgment with peer
review central; use quantitative indicators responsibly; abandon
inappropriate uses of journal- and publication-based measures, especially
Journal Impact Factor and h-index; publish criteria; train assessors; review
and evaluate criteria, tools, and processes.
- **Use here:** qualitative-first process, rubric provenance, rater training,
monitoring, and no metric shortcut.
### Hong Kong Principles
- **Primary article:** Moher et al. (2020), [“The Hong Kong Principles for
assessing researchers: Fostering research
integrity”](https://doi.org/10.1371/journal.pbio.3000737), *PLOS Biology*
18(7):e3000737.
- **Published:** 2020-07-16.
- **Official implementation page:** [World Conferences on Research Integrity
Foundation](https://www.wcrif.org/guidance/hong-kong-principles)
- **Verified points:** assess responsible practices, value complete reporting,
reward open research, acknowledge diverse research activity, and recognize
essential work such as review and mentoring.
- **Use here:** integrity, reporting, appropriate openness, and diverse
contribution evidence. The principles do not supply validated score weights.
### The Metric Tide and its commissioned revisit
- **Primary public-sector source:** Research England/UKRI, [*The Metric
Tide*](https://www.ukri.org/publications/review-of-metrics-in-research-assessment-and-management/).
- **Published:** 2015-07-06.
- **Revisit:** Curry, Gadd, and Wilsdon, [*Harnessing the Metric
Tide*](https://rori.figshare.com/articles/report/Harnessing_the_Metric_Tide/21701624),
version 2 (DOI 10.6084/m9.figshare.21701624).
- **Posted:** 2022-12-12; commissioned by the joint UK higher-education funding
bodies for the Future Research Assessment Programme.
- **Verified points:** the revisit recommends putting principles into practice,
evaluating with those evaluated, avoiding all-metric approaches, using data
for public benefit, and rethinking rankings.
- **Status limitation:** *Harnessing the Metric Tide* describes itself as an
independent input to deliberations, not the eventual policy conclusion.
- **Use here:** stakeholder participation, no all-metric process, and explicit
scrutiny of rankings and system effects.
### Current UKRI guidance
- **Primary policy:** [UKRI funding assessment and decision-making policy and
principles](https://www.ukri.org/publications/ukri-principles-of-assessment-and-decision-making/uk-research-and-innovation-ukri-funding-assessment-and-decision-making-policy-and-principles)
- **Primary implementation guidance:** [Résumé for Research and Innovation
(R4RI)](https://www.ukri.org/apply-for-funding/develop-your-application/resume-for-research-and-innovation-r4ri-guidance/)
- **R4RI last updated:** 2026-04-30.
- **Verified points:** UKRI will not use journal-based measures as surrogates
for article quality, individual contribution, or funding decisions. R4RI
evidences a wider range of team contributions; assessors do not score its
individual modules or view it in isolation.
- **Use here:** diverse contribution evidence and contextual review. This skill
nevertheless blocks funding decisions entirely; the UKRI material is
guidance context, not authorization to support such decisions.
### UNESCO Recommendation on Open Science
- **Primary source:** [UNESCO Recommendation on Open
Science](https://unesdoc.unesco.org/ark:/48223/pf0000379949).
- **Adopted:** 2021-11-23 by the UNESCO General Conference.
- **Official overview:** [UNESCO Open Science](https://www.unesco.org/en/open-science/about)
- **Verified points:** quality and integrity, collective benefit, equity,
fairness, diversity, inclusion, open engagement, training, and incentives
aligned with open science; open science must not leave people, languages,
disciplines, or knowledge systems behind.
- **Use here:** assess responsible openness in context. Privacy, safety,
consent, sovereignty, and legitimate restrictions can outweigh openness.
### INORMS SCOPE framework
- **Primary source:** [SCOPE Framework full guide,
v1.0](https://inorms.net/wp-content/uploads/2022/03/21655-scope-guide-v10.pdf).
- **Current official page:** [INORMS SCOPE Framework for Research
Evaluation](https://inorms.net/scope-framework-for-research-evaluation).
- **Verified points:** Start with values; consider Context; identify Options;
Probe for discrimination, gaming, unintended effects, and cost-benefit; and
Evaluate the evaluation. Evaluate only where needed, with those evaluated,
and with evaluation expertise.
- **Use here:** process design and the bias/process checklist.
### CRediT contributor taxonomy
- **Primary source:** [CRediT](https://credit.niso.org/).
- **Standard:** ANSI/NISO Z39.104-2022, approved 2022-01-14 and published
2022-02-08.
- **Verified points:** 14 roles provide transparent attribution of diverse
contributions. CRediT does not determine authorship or contribution quality.
- **Use here:** optional vocabulary for contribution evidence, never a score.
## Measurement, fairness, accessibility, and privacy
### Standards for Educational and Psychological Testing
- **Primary source:** AERA, APA, and NCME, [*Standards for Educational and
Psychological Testing*, 2014
edition](https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf).
- **Official status page:** [NCME Testing Standards](https://ncme.org/resources/books/testing-standards/).
- **Status:** NCME still links the open-access 2014 edition. The APA page was
not readable during this refresh; no later completed edition or current
revision timetable was verified.
- **Verified points:** intended interpretations and uses require validity
evidence; reliability/precision and relevant errors should be reported;
rater selection, training, qualification, monitoring, agreement, accuracy,
and drift need documentation; fairness and subgroup validity require
evidence; uncertainty should accompany estimates.
- **Use here:** these are measurement principles, not proof that this rubric is
a psychological test. The template records evidence gaps and must not be
described as validated psychometrics.
### Accessibility
- **Primary source:** W3C, [Web Content Accessibility Guidelines
2.2](https://www.w3.org/TR/WCAG22/).
- **Status:** W3C Recommendation published 2023-10-05; update noted
2024-12-12.
- **Use here:** accessible materials and reasonable accommodation processes
are required; local legal and institutional requirements may be broader.
### Data protection
- **Primary guidance:** UK Information Commissioner's Office,
[purpose limitation](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-protection-principles/a-guide-to-the-data-protection-principles/purpose-limitation)
and [data minimisation](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-protection-principles/a-guide-to-the-data-protection-principles/data-minimisation).
- **Current guidance dates found:** purpose limitation updated 2026-03-23;
data minimisation page published 2025-09-09.
- **Verified points:** specify legitimate purposes and process only adequate,
relevant, necessary data; review and delete data no longer needed.
- **Use here:** scripts accept only minimized IDs, scores, statuses, and local
references. They reject common private-application fields and never reproduce
raw source documents.
## ScholarEval paper and project status
- **Exact paper:** Hanane Nour Moussa, Patrick Queiroz Da Silva, Daniel
Adu-Ampratwum, Alyson East, Zitong Lu, Nikki Puccetti, Mingyi Xue, Huan Sun,
Bodhisattwa Prasad Majumder, and Sachin Kumar, [*ScholarEval: Research Idea
Evaluation Grounded in Literature*](https://arxiv.org/abs/2510.16234).
- **Verified status:** arXiv:2510.16234, submitted 2025-10-17; latest verified
version **v2**, revised 2026-02-28. The displayed DOI
`10.48550/arXiv.2510.16234` is an arXiv/DataCite DOI, not evidence of journal
publication.
- **Official project:** [skai-research/ScholarEval](https://github.com/skai-research/ScholarEval).
The repository describes itself as official code and data and cites the work
as `@misc`; no release or peer-reviewed publication claim was verified.
- **Review-status caution:** the [OpenReview forum](https://openreview.net/forum?id=0TPKIbihMF)
returned a browser-verification challenge on 2026-10-01. Its indexed submission
PDF says under review; that historical PDF does not establish a current
decision. ICLR pages were inaccessible to this review. Acceptance/peer-reviewed
publication remains unverified, not disproven.
- **What the preprint reports:** a retrieval-augmented framework assessing
research ideas for soundness and contribution; a 117-idea, four-discipline
dataset; coverage comparisons against expert-annotated review points; and a
user study.
- **What it does not establish:** validated psychometric measurement of
scholar quality, transportability to personnel or funding decisions, validity
of this skill's generalized rubric, stable cross-discipline score meaning, or
freedom from subgroup bias.
## Work-version and correction metadata
- **Primary source:** [Crossmark](https://www.crossref.org/services/crossmark/).
- **Verified points:** publishers deposit changes and correction/retraction
information; Crossref explicitly says the presence of Crossmark is not a
guarantee. Missing metadata cannot establish absence of an update.
- **Use here:** preserve exact version/access date, inspect available publisher
notices, record unresolved status, and issue a fresh evaluation after changes.
An identifier or reachable URL does not prove that a cited source supports
a claim. Read the relevant content and separate metadata verification from
substantive evidence review.
## Review cadence
Re-check this ledger before any rubric adoption and at least annually. Re-check
the ScholarEval arXiv and official project records before describing its
publication status. Record any local disciplinary standards separately; a
global source cannot substitute for local construct validation.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |