Skip to content

K-Dense-AI/scientific-agent-skills / skills/scholar-evaluation

scholar-evaluation

Verified

Provides qualitative-first, evidence-traceable developmental review of scholarly works and audit low-stakes research-assessment rubrics with optional local quality controls. Never use for ranking people or consequential decisions.

references/source_ledger.md

View on GitHub
# Dated Source Ledger

Refreshed on **2026-10-01** against the current official pages, source PDFs,
and ScholarEval project record below. This updates the 2026-07-23 review.
Policy-source verification and local synthetic-tool tests are distinct: no
researcher profiles, bibliometric datasets, authenticated services, external
model pipeline, or private work were evaluated.

The bundled workflow is local JSON/CSV only. It contains no scholarly-database,
author-identity, citation-count, authentication, pagination, or rate-limit API
contract. The shared collection citation's arXiv request is a metadata lookup,
not an evaluation service. Do not graft provider endpoints into this rubric.

## Responsible research assessment

### San Francisco Declaration on Research Assessment (DORA)

- **Primary source:** [Read the Declaration](https://sfdora.org/read/)
- **Origin:** developed in 2012; official page rechecked 2026-10-01.
- **Verified points:** assess work on its own merits; do not use journal-based
  measures as surrogates for an article or a person's contribution; state
  criteria explicitly; consider data, software, and other outputs; use
  qualitative evidence; make metric methods transparent and account for field
  and output-type variation.
- **Use here:** categorical ban on scored prestige proxies and a requirement
  for explicit criteria, diverse evidence, and traceability.

### DORA quantitative-indicator guidance

- **Primary source:** [Guidance on the responsible use of quantitative
  indicators](https://sfdora.org/resource/guidance-on-the-responsible-use-of-quantitative-indicators-in-research-assessment/)
- **Document:** [official PDF](https://sfdora.org/wp-content/uploads/2024/05/DORA_indicators_guidance.pdf)
- **Release:** 2024; official resource page accessible in 2026.
- **Verified points:** no indicator captures research quality in one number;
  uses should be clear, transparent, specific, contextual, and fair. The
  guidance addresses journal measures, citation counts, the h-index,
  field-normalized indicators, and altmetrics; it warns about reductive,
  aggregate, composite, lagging, field, career-stage, and bias effects.
- **Use here:** quantitative indicators are excluded from rubric scores. If
  mentioned descriptively outside the tools, their purpose, data, coverage,
  time window, field normalization, uncertainty, bias, and non-quality meaning
  must be explicit.

### Leiden Manifesto

- **Primary source:** Hicks, Wouters, Waltman, de Rijcke, and Rafols,
  [“Bibliometrics: The Leiden Manifesto for research
  metrics”](https://doi.org/10.1038/520429a), *Nature* 520, 429–431.
- **Published:** 2015-04-22. Publisher metadata remains current. The publisher
  full text was access-limited during this refresh; principles were checked
  against the [author-linked Spanish translation](https://www.leidenmanifesto.org/uploads/4/1/6/0/41603901/manifiesto_cast.pdf)
  hosted by the official manifesto site.
- **Verified points:** quantitative evaluation should support qualitative
  expert assessment; measure against missions; protect locally relevant
  research; account for field variation; keep data and analysis open and
  verifiable; allow those evaluated to verify data; account for career age and database differences;
  avoid false precision; recognize gaming and system effects; review indicators
  regularly.
- **Use here:** contextualization, inspectability, uncertainty, fairness review,
  and periodic revision.

### Agreement on Reforming Research Assessment / CoARA

- **Primary record:** [Agreement on Reforming Research
  Assessment](https://doi.org/10.5281/zenodo.13480728), version 1.
- **Published:** 2022-07-20; Zenodo record modified 2024-08-29.
- **Current official overview:** [CoARA Agreement](https://www.coara.org/agreement/the-agreement-full-text/)
- **Verified points:** recognize diverse outputs, practices, activities, roles,
  and careers; base assessment primarily on qualitative judgment with peer
  review central; use quantitative indicators responsibly; abandon
  inappropriate uses of journal- and publication-based measures, especially
  Journal Impact Factor and h-index; publish criteria; train assessors; review
  and evaluate criteria, tools, and processes.
- **Use here:** qualitative-first process, rubric provenance, rater training,
  monitoring, and no metric shortcut.

### Hong Kong Principles

- **Primary article:** Moher et al. (2020), [“The Hong Kong Principles for
  assessing researchers: Fostering research
  integrity”](https://doi.org/10.1371/journal.pbio.3000737), *PLOS Biology*
  18(7):e3000737.
- **Published:** 2020-07-16.
- **Official implementation page:** [World Conferences on Research Integrity
  Foundation](https://www.wcrif.org/guidance/hong-kong-principles)
- **Verified points:** assess responsible practices, value complete reporting,
  reward open research, acknowledge diverse research activity, and recognize
  essential work such as review and mentoring.
- **Use here:** integrity, reporting, appropriate openness, and diverse
  contribution evidence. The principles do not supply validated score weights.

### The Metric Tide and its commissioned revisit

- **Primary public-sector source:** Research England/UKRI, [*The Metric
  Tide*](https://www.ukri.org/publications/review-of-metrics-in-research-assessment-and-management/).
- **Published:** 2015-07-06.
- **Revisit:** Curry, Gadd, and Wilsdon, [*Harnessing the Metric
  Tide*](https://rori.figshare.com/articles/report/Harnessing_the_Metric_Tide/21701624),
  version 2 (DOI 10.6084/m9.figshare.21701624).
- **Posted:** 2022-12-12; commissioned by the joint UK higher-education funding
  bodies for the Future Research Assessment Programme.
- **Verified points:** the revisit recommends putting principles into practice,
  evaluating with those evaluated, avoiding all-metric approaches, using data
  for public benefit, and rethinking rankings.
- **Status limitation:** *Harnessing the Metric Tide* describes itself as an
  independent input to deliberations, not the eventual policy conclusion.
- **Use here:** stakeholder participation, no all-metric process, and explicit
  scrutiny of rankings and system effects.

### Current UKRI guidance

- **Primary policy:** [UKRI funding assessment and decision-making policy and
  principles](https://www.ukri.org/publications/ukri-principles-of-assessment-and-decision-making/uk-research-and-innovation-ukri-funding-assessment-and-decision-making-policy-and-principles)
- **Primary implementation guidance:** [Résumé for Research and Innovation
  (R4RI)](https://www.ukri.org/apply-for-funding/develop-your-application/resume-for-research-and-innovation-r4ri-guidance/)
- **R4RI last updated:** 2026-04-30.
- **Verified points:** UKRI will not use journal-based measures as surrogates
  for article quality, individual contribution, or funding decisions. R4RI
  evidences a wider range of team contributions; assessors do not score its
  individual modules or view it in isolation.
- **Use here:** diverse contribution evidence and contextual review. This skill
  nevertheless blocks funding decisions entirely; the UKRI material is
  guidance context, not authorization to support such decisions.

### UNESCO Recommendation on Open Science

- **Primary source:** [UNESCO Recommendation on Open
  Science](https://unesdoc.unesco.org/ark:/48223/pf0000379949).
- **Adopted:** 2021-11-23 by the UNESCO General Conference.
- **Official overview:** [UNESCO Open Science](https://www.unesco.org/en/open-science/about)
- **Verified points:** quality and integrity, collective benefit, equity,
  fairness, diversity, inclusion, open engagement, training, and incentives
  aligned with open science; open science must not leave people, languages,
  disciplines, or knowledge systems behind.
- **Use here:** assess responsible openness in context. Privacy, safety,
  consent, sovereignty, and legitimate restrictions can outweigh openness.

### INORMS SCOPE framework

- **Primary source:** [SCOPE Framework full guide,
  v1.0](https://inorms.net/wp-content/uploads/2022/03/21655-scope-guide-v10.pdf).
- **Current official page:** [INORMS SCOPE Framework for Research
  Evaluation](https://inorms.net/scope-framework-for-research-evaluation).
- **Verified points:** Start with values; consider Context; identify Options;
  Probe for discrimination, gaming, unintended effects, and cost-benefit; and
  Evaluate the evaluation. Evaluate only where needed, with those evaluated,
  and with evaluation expertise.
- **Use here:** process design and the bias/process checklist.

### CRediT contributor taxonomy

- **Primary source:** [CRediT](https://credit.niso.org/).
- **Standard:** ANSI/NISO Z39.104-2022, approved 2022-01-14 and published
  2022-02-08.
- **Verified points:** 14 roles provide transparent attribution of diverse
  contributions. CRediT does not determine authorship or contribution quality.
- **Use here:** optional vocabulary for contribution evidence, never a score.

## Measurement, fairness, accessibility, and privacy

### Standards for Educational and Psychological Testing

- **Primary source:** AERA, APA, and NCME, [*Standards for Educational and
  Psychological Testing*, 2014
  edition](https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf).
- **Official status page:** [NCME Testing Standards](https://ncme.org/resources/books/testing-standards/).
- **Status:** NCME still links the open-access 2014 edition. The APA page was
  not readable during this refresh; no later completed edition or current
  revision timetable was verified.
- **Verified points:** intended interpretations and uses require validity
  evidence; reliability/precision and relevant errors should be reported;
  rater selection, training, qualification, monitoring, agreement, accuracy,
  and drift need documentation; fairness and subgroup validity require
  evidence; uncertainty should accompany estimates.
- **Use here:** these are measurement principles, not proof that this rubric is
  a psychological test. The template records evidence gaps and must not be
  described as validated psychometrics.

### Accessibility

- **Primary source:** W3C, [Web Content Accessibility Guidelines
  2.2](https://www.w3.org/TR/WCAG22/).
- **Status:** W3C Recommendation published 2023-10-05; update noted
  2024-12-12.
- **Use here:** accessible materials and reasonable accommodation processes
  are required; local legal and institutional requirements may be broader.

### Data protection

- **Primary guidance:** UK Information Commissioner's Office,
  [purpose limitation](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-protection-principles/a-guide-to-the-data-protection-principles/purpose-limitation)
  and [data minimisation](https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/data-protection-principles/a-guide-to-the-data-protection-principles/data-minimisation).
- **Current guidance dates found:** purpose limitation updated 2026-03-23;
  data minimisation page published 2025-09-09.
- **Verified points:** specify legitimate purposes and process only adequate,
  relevant, necessary data; review and delete data no longer needed.
- **Use here:** scripts accept only minimized IDs, scores, statuses, and local
  references. They reject common private-application fields and never reproduce
  raw source documents.

## ScholarEval paper and project status

- **Exact paper:** Hanane Nour Moussa, Patrick Queiroz Da Silva, Daniel
  Adu-Ampratwum, Alyson East, Zitong Lu, Nikki Puccetti, Mingyi Xue, Huan Sun,
  Bodhisattwa Prasad Majumder, and Sachin Kumar, [*ScholarEval: Research Idea
  Evaluation Grounded in Literature*](https://arxiv.org/abs/2510.16234).
- **Verified status:** arXiv:2510.16234, submitted 2025-10-17; latest verified
  version **v2**, revised 2026-02-28. The displayed DOI
  `10.48550/arXiv.2510.16234` is an arXiv/DataCite DOI, not evidence of journal
  publication.
- **Official project:** [skai-research/ScholarEval](https://github.com/skai-research/ScholarEval).
  The repository describes itself as official code and data and cites the work
  as `@misc`; no release or peer-reviewed publication claim was verified.
- **Review-status caution:** the [OpenReview forum](https://openreview.net/forum?id=0TPKIbihMF)
  returned a browser-verification challenge on 2026-10-01. Its indexed submission
  PDF says under review; that historical PDF does not establish a current
  decision. ICLR pages were inaccessible to this review. Acceptance/peer-reviewed
  publication remains unverified, not disproven.
- **What the preprint reports:** a retrieval-augmented framework assessing
  research ideas for soundness and contribution; a 117-idea, four-discipline
  dataset; coverage comparisons against expert-annotated review points; and a
  user study.
- **What it does not establish:** validated psychometric measurement of
  scholar quality, transportability to personnel or funding decisions, validity
  of this skill's generalized rubric, stable cross-discipline score meaning, or
  freedom from subgroup bias.

## Work-version and correction metadata

- **Primary source:** [Crossmark](https://www.crossref.org/services/crossmark/).
- **Verified points:** publishers deposit changes and correction/retraction
  information; Crossref explicitly says the presence of Crossmark is not a
  guarantee. Missing metadata cannot establish absence of an update.
- **Use here:** preserve exact version/access date, inspect available publisher
  notices, record unresolved status, and issue a fresh evaluation after changes.
  An identifier or reachable URL does not prove that a cited source supports
  a claim. Read the relevant content and separate metadata verification from
  substantive evidence review.

## Review cadence

Re-check this ledger before any rubric adoption and at least annually. Re-check
the ScholarEval arXiv and official project records before describing its
publication status. Record any local disciplinary standards separately; a
global source cannot substitute for local construct validation.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall