DAAF-Contribution-Community/daaf / .claude/skills/scholarly-search
scholarly-search
FlaggedSearches and traces academic literature via DAAF's provenance CLI `bash /daaf/scripts/scholar_search.sh`, fusing OpenAlex, Semantic Scholar and ERIC (incl. EdWorkingPapers), merging paper versions, and saving raw responses, a fenced digest and a MANIFEST row for every call. Subcommands: citation verification (verify: catches fabricated references), forward/backward snowballing, more-like-this, passages inside OA papers, cached full text, legal OA copies and licenses (oa), retraction checks (Crossref/Retraction Watch, Scite notices), Scite citation-stance tallies (supporting/contradicting), and Elicit study-type-tagged search and paid async research reports (spend-gated). Use for literature reviews, finding papers or evidence on a question, checking that a citation exists, was retracted, or is supported or contradicted by later work, or locating an OA PDF. Strong for education and social science. Grey literature goes through the web-retrieval skill. NOT for dataset discovery (education-data-explorer).
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 44.1 KB |
| references/_BACKEND_TEMPLATE.md | markdown | 9.1 KB |
| references/adding-a-backend.md | markdown | 15.1 KB |
| references/api-notes.md | markdown | 144.5 KB |
| references/citation-adjudication.md | markdown | 36.5 KB |
| references/crossref.md | markdown | 4.0 KB |
| references/elicit.md | markdown | 30.5 KB |
| references/eric.md | markdown | 3.4 KB |
| references/licensing.md | markdown | 6.5 KB |
| references/openalex.md | markdown | 13.4 KB |
| references/scite.md | markdown | 11.5 KB |
| references/semantic-scholar.md | markdown | 7.1 KB |
| references/unpaywall.md | markdown | 4.6 KB |
references/unpaywall.md
View on GitHub# Unpaywall
DOI-in, open-access-answer-out. Since late 2025 Unpaywall runs on the OpenAlex
database: its `is_oa`, `oa_status` and PDF link are the same values as an
OpenAlex work's `open_access` and `best_oa_location`
(help.openalex.org/access/unpaywall, fetched 2026-10-02; confirmed live
2026-10-02, below). In `oa` it is therefore a **fallback leg** after
OpenAlex. Adapter: `scripts/scholar_backends/unpaywall.py`.
## Auth
No key, but every request needs `?email=`. The tool uses the user's
`SCHOLARLY_CONTACT_EMAIL` and nothing else: unset → the leg is **skipped with
a note** (`unpaywall: skipped: SCHOLARLY_CONTACT_EMAIL not set`). The address
is sent only as that parameter and is replaced by `[SCHOLARLY_CONTACT_EMAIL]`
everywhere the tool writes or prints: logged params, the raw `.json`
(including any redirect chain or echoed error body), the digest, the MANIFEST
row, stdout and stderr (`scrub_contact()` in the core).
## Endpoints used
| Use | Request | Command |
|-----|---------|---------|
| OA record | `GET https://api.unpaywall.org/v2/{doi}?email=…` | `oa` |
Response fields used: `doi`, `title`, `year`, `journal_name`, `genre`,
`z_authors[].raw_author_name` (or `given`/`family`), `is_oa`, `oa_status`,
`best_oa_location{url, url_for_pdf, url_for_landing_page, license, version,
host_type, repository_institution}`, `oa_locations[]`.
## Cost and limits
Free. A 100,000 calls/day figure circulates in search snippets but is
**unverified**: unpaywall.org's API and data-format pages are JS-rendered and
produced no extract on 2026-10-02, and the live responses carry no
rate-limit headers. The tool gates the lane at 0.1 s.
## Measured behavior
**2026-10-02, live** — `oa --all-legs` on three DOIs,
`DAAF_FETCH_CONTEXT=FrameworkDev_S1_unpaywall_live`. The unit-test fixtures
`tests/python/fixtures/scholar_search/unpaywall_{record,closed,404}.json`
are trimmed from that run's raw response, which is not in this repository.
| DOI | HTTP | Observed |
|-----|------|----------|
| `10.1371/journal.pmed.0020124` (PLOS, gold) | 200 | `oa_status: gold`; `best_oa_location` is the publisher **landing page** (`url_for_pdf: null`, `version: publishedVersion`, `license: cc-by`); the PDF is in a `repository` location (RePEc, `submittedVersion`). 6 `oa_locations` (two share a PMC URL). Matches OpenAlex's `best_oa_location` for the same DOI in the same run. |
| `10.1093/qje/qjv036` (QJE, closed) | 200 | `is_oa: false`, `oa_status: closed`, `best_oa_location: null`, `oa_locations: []` — closed is a 200 answer, not a 404. |
| `10.48550/arxiv.2205.01833` (DataCite/arXiv) | 404 | HTML body (a generic "404 Not Found" page), not JSON — confirms "Crossref DOIs only". |
**Doc-derived shape vs live** (what the old fixture got wrong):
- `z_authors` entries carry `raw_author_name`, `author_position`,
`is_corresponding` and `raw_affiliation_strings` — no `given` / `family`.
The mapper falls back to `raw_author_name`, so author names were never
lost.
- The best location had no `url_for_pdf` here; the doc-derived fixture put a
PDF on the publisher best location.
- Extra top-level fields not in the doc summary: `first_oa_location`,
`is_paratext`, `data_standard` (2), `has_repository_copy`,
`journal_is_in_doaj`, `journal_is_oa`, `journal_issn_l`, `journal_issns`,
`oa_locations_embargoed`, `published_date`, `publisher`, `updated`.
Location-level `evidence` and `updated` both read `"deprecated"`.
- `host_type` is always present, so the adapter's old
`host_type or repository_institution` fallback never reached the
institution name; candidates now report `repository: <institution>`.
- The response does **not** echo the email (the raw artifact held the
placeholder only in the three request params).
## Storage and ToS
Same data as OpenAlex (CC0); `LICENSE_CLASS = "open"`. The PDFs it points to
keep their own copyright: record `license` and `version` and redistribute only
CC BY / BY-SA / CC0 items.
## Exit mapping
Never changes `oa`'s exit code on its own:
- unconfigured (no `SCHOLARLY_CONTACT_EMAIL`) → `unpaywall: skipped: …`
(never exit 10);
- HTTP 404 → `unpaywall: not found (HTTP 404: DOI unknown to Unpaywall,
which covers Crossref DOIs only)` — an answer, and `oa` continues to
ERIC/arXiv;
- closed paper → HTTP 200 with `unpaywall: 0 location(s)`;
- other HTTP errors after retries, network failure or timeout →
`unpaywall: failed: …` and `oa` continues to ERIC/arXiv.
## Minimal smoke test
With `SCHOLARLY_CONTACT_EMAIL` set host-side in `environment_settings.txt`
(recreate the container after editing it):
```
bash /daaf/scripts/scholar_search.sh oa 10.1371/journal.pmed.0020124 {PROJECT_DIR}/lit_search --all-legs
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| canvas-designanthropics/skillsCreate beautiful visual art in .png and .pdf documents using design philosophy. You should use this skill when the user asks to create a poster, piece of art, design, or other static piece. Create original visual designs, never copying existing artists' work to avoid copyright violations. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| docxanthropics/skillsUse this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files) or Word templates (.dotx files). Triggers include: any mention of 'Word doc', 'word document', '.docx', '.dotx', or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting or reorganizing content from .docx or .dotx files, inserting or replacing images in documents, performing find-and-replace in Word files, working with tracked changes or comments, or converting content into a polished Word document. If the user asks for a 'report', 'memo', 'letter', 'template', or similar deliverable as a Word or .docx file, use this skill. Do NOT use for PDFs, spreadsheets, Google Docs, or general coding tasks unrelated to document generation. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| pdfanthropics/skillsUse this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text/tables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting/decrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| pptxanthropics/skillsUse this skill any time a .pptx or .potx file is involved in any way — as input, output, or both. This includes: creating slide decks, pitch decks, or presentations; reading, parsing, or extracting text from any .pptx or .potx file (even if the extracted content will be used elsewhere, like in an email or summary); editing, modifying, or updating existing presentations; combining or splitting slide files; working with templates (.potx), layouts, speaker notes, or comments. Trigger whenever the user mentions "deck," "slides," "presentation," or references a .pptx or .potx filename, regardless of what they plan to do with the content afterward. If a .pptx or .potx file needs to be opened, created, or touched, use this skill. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |