Skip to content

K-Dense-AI/scientific-agent-skills / skills/citation-management

citation-management

Verified

Comprehensive citation management for academic research. Search OpenAlex, PubMed, and Google Scholar for papers, extract accurate metadata, validate citations, and generate properly formatted BibTeX entries. This skill should be used when you need to find papers, verify citation information, convert DOIs to BibTeX, or ensure reference accuracy in scientific writing.

references/pubmed_search.md

View on GitHub
# PubMed Search Guide

Comprehensive guide to searching PubMed for biomedical and life sciences literature, including MeSH terms, field tags, advanced search strategies, and E-utilities API usage.

## Overview

PubMed is the premier database for biomedical literature:
- **Coverage**: Biomedical citations, including MEDLINE records and online books
- **Scope**: Biomedical and life sciences
- **Sources**: MEDLINE, life science journals, online books
- **Authority**: Maintained by National Library of Medicine (NLM) / NCBI
- **Access**: Free, no account required
- **Updates**: Daily with new citations
- **Curation**: High-quality metadata, MeSH indexing

## Basic Search

### Simple Keyword Search

PubMed automatically maps terms to MeSH and searches multiple fields:

```
diabetes
CRISPR gene editing
Alzheimer's disease treatment
cancer immunotherapy
```

**Automatic Features**:
- Automatic MeSH mapping
- Plural/singular variants
- Abbreviation expansion
- Spell checking

### Exact Phrase Search

Use quotation marks for exact phrases:

```
"CRISPR-Cas9"
"systematic review"
"randomized controlled trial"
"machine learning"
```

## MeSH (Medical Subject Headings)

### What is MeSH?

MeSH is a controlled vocabulary thesaurus for indexing biomedical literature:
- **Hierarchical structure**: Organized in tree structures
- **Consistent indexing**: Same concept always tagged the same way
- **Comprehensive**: Covers diseases, drugs, anatomy, techniques, etc.
- **Indexing**: NLM assigns MeSH using automated indexing and curation; recent and non-MEDLINE records can lack MeSH

### Finding MeSH Terms

**MeSH Browser**: https://meshb.nlm.nih.gov/search

**Example**:
```
Search: "heart attack"
MeSH term: "Myocardial Infarction"
```

**In PubMed**:
1. Search with keyword
2. Open a relevant article record and inspect its MeSH terms when present
3. Confirm descriptors in the MeSH Browser
4. Combine descriptors with title/abstract synonyms

### Using MeSH in Searches

**Basic MeSH search**:
```
"Diabetes Mellitus"[MeSH]
"CRISPR-Cas Systems"[MeSH]
"Alzheimer Disease"[MeSH]
"Neoplasms"[MeSH]
```

**MeSH with subheadings**:
```
"Diabetes Mellitus/drug therapy"[MeSH]
"Neoplasms/genetics"[MeSH]
"Heart Failure/prevention and control"[MeSH]
```

**Common subheadings**:
- `/drug therapy`: Drug treatment
- `/diagnosis`: Diagnostic aspects
- `/genetics`: Genetic aspects
- `/epidemiology`: Occurrence and distribution
- `/prevention and control`: Prevention methods
- `/etiology`: Causes
- `/surgery`: Surgical treatment
- `/metabolism`: Metabolic aspects

### MeSH Explosion

By default, MeSH searches include narrower terms (explosion):

```
"Neoplasms"[MeSH]
# Includes: Breast Neoplasms, Lung Neoplasms, etc.
```

**Disable explosion** (exact term only):
```
"Neoplasms"[MeSH:NoExp]
```

### MeSH Major Topic

Search only where MeSH term is a major focus:

```
"Diabetes Mellitus"[MeSH Major Topic]
# Only papers where diabetes is main topic
```

## Field Tags

Field tags specify which part of the record to search.

### Common Field Tags

**Title and Abstract**:
```
cancer[Title]                    # In title only
treatment[Title/Abstract]        # In title or abstract
"machine learning"[Title/Abstract]
```

**Author**:
```
"Smith J"[Author]
"Doudna JA"[Author]
"Collins FS"[Author]
```

**Author - Full Name**:
```
"Smith, John"[Full Author Name]
```

**Journal**:
```
"Nature"[Journal]
"Science"[Journal]
"New England Journal of Medicine"[Journal]
"Nat Commun"[Journal]           # Abbreviated form
```

**Publication Date**:
```
2023[Publication Date]
2020:2024[Publication Date]      # Date range
2023/01/01:2023/12/31[Publication Date]
```

**Date Created**:
```
2023[Date - Create]              # When added to PubMed
```

**Publication Type**:
```
"Review"[Publication Type]
"Clinical Trial"[Publication Type]
"Meta-Analysis"[Publication Type]
"Randomized Controlled Trial"[Publication Type]
```

**Language**:
```
English[Language]
French[Language]
```

**DOI**:
```
10.1038/nature12345[DOI]
```

**PMID (PubMed ID)**:
```
12345678[PMID]
```

**Article ID**:
```
PMC1234567[PMC]                  # PubMed Central ID
```

### Less Common But Useful Tags

```
humans[MeSH Terms]               # Only human studies
animals[MeSH Terms]              # Only animal studies
"United States"[Place of Publication]
nih[Grant Number]                # NIH-funded research
"Female"[MeSH Terms]                    # Female subjects
"Aged, 80 and over"[MeSH Terms]        # Elderly subjects
```

## Boolean Operators

Combine search terms with Boolean logic.

### AND

Both terms must be present (default behavior):

```
diabetes AND treatment
"CRISPR-Cas9" AND "gene editing"
cancer AND immunotherapy AND "clinical trial"[Publication Type]
```

### OR

Either term must be present:

```
"heart attack" OR "myocardial infarction"
diabetes OR "diabetes mellitus"
CRISPR OR Cas9 OR "gene editing"
```

**Use case**: Synonyms and related terms

### NOT

Exclude terms:

```
cancer NOT review
diabetes NOT animal
"machine learning" NOT "deep learning"
```

**Caution**: May exclude relevant papers that mention both terms.

### Combining Operators

Use parentheses for complex logic:

```
(diabetes OR "diabetes mellitus") AND (treatment OR therapy)

("CRISPR" OR "gene editing") AND ("therapeutic" OR "therapy") 
  AND 2020:2024[Publication Date]

(cancer OR neoplasm) AND (immunotherapy OR "immune checkpoint inhibitor") 
  AND ("clinical trial"[Publication Type] OR "randomized controlled trial"[Publication Type])
```

## Advanced Search Builder

**Access**: https://pubmed.ncbi.nlm.nih.gov/advanced/

**Features**:
- Visual query builder
- Add multiple query boxes
- Select field tags from dropdowns
- Combine with AND/OR/NOT
- Preview results
- Shows final query string
- Save queries

**Workflow**:
1. Add search terms in separate boxes
2. Select field tags
3. Choose Boolean operators
4. Preview results
5. Refine as needed
6. Copy final query string
7. Use in scripts or save

**Example built query**:
```
#1: "Diabetes Mellitus, Type 2"[MeSH]
#2: "Metformin"[MeSH]
#3: "Clinical Trial"[Publication Type]
#4: 2020:2024[Publication Date]
#5: #1 AND #2 AND #3 AND #4
```

## Filters and Limits

### Article Types

```
"Review"[Publication Type]
"Systematic Review"[Publication Type]
"Meta-Analysis"[Publication Type]
"Clinical Trial"[Publication Type]
"Randomized Controlled Trial"[Publication Type]
"Case Reports"[Publication Type]
"Comparative Study"[Publication Type]
```

### Species

```
humans[MeSH Terms]
mice[MeSH Terms]
rats[MeSH Terms]
```

### Sex

```
"Female"[MeSH Terms]
"Male"[MeSH Terms]
```

### Age Groups

```
"Infant"[MeSH Terms]
"Child"[MeSH Terms]
"Adolescent"[MeSH Terms]
"Adult"[MeSH Terms]
"Aged"[MeSH Terms]
"Aged, 80 and over"[MeSH Terms]
```

### Text Availability

```
free full text[Filter]           # Free full-text available
```

### Journal Categories

```
"Journal Article"[Publication Type]
```

## E-utilities API

NCBI provides programmatic access via E-utilities (Entrez Programming Utilities).

### Overview

**Base URL**: `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/`

**Main Tools**:
- **ESearch**: Search and retrieve PMIDs
- **EFetch**: Retrieve full records
- **ESummary**: Retrieve document summaries
- **ELink**: Find related articles
- **EInfo**: Database statistics

**No API key required**, but recommended for:
- Higher rate limits (10/sec vs 3/sec)
- Better performance
- Identify your project

**Get API key**: https://www.ncbi.nlm.nih.gov/account/

### ESearch - Search PubMed

Retrieve PMIDs for a query. The 10,000 ceiling is a total query window, not
a per-page limit that cursor/history parameters bypass. Split larger searches
into disjoint date ranges and deduplicate, or use NCBI EDirect. See
[NCBI's PubMed update](https://ncbiinsights.ncbi.nlm.nih.gov/2021/10/05/updated-pubmed-api/).

**Endpoint**: `/esearch.fcgi`

**Parameters**:
- `db`: Database (pubmed)
- `term`: Search query
- `retmax`: Maximum results (default 20, max 10000)
- `retstart`: Zero-based offset within the first 10,000 matches; `retstart + retmax` cannot exceed 10,000
- `sort`: PubMed values include `relevance`, `pub_date`, `Author`, and `JournalName`
- `api_key`: Your API key (optional but recommended)

**Example URL**:
```
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?
  db=pubmed&
  term=diabetes+AND+treatment&
  retmax=100&
  retmode=json&
  api_key=YOUR_API_KEY
```

**Illustrative response (two-record excerpt)**:
```json
{
  "esearchresult": {
    "count": "250000",
    "retmax": "100",
    "idlist": ["12345678", "12345679"]
  }
}
```

### EFetch - Retrieve Records

Get full metadata for PMIDs.

**Endpoint**: `/efetch.fcgi`

**Parameters**:
- `db`: Database (pubmed)
- `id`: Comma-separated PMIDs
- `retmode=xml`: Full PubMed XML (omit `rettype`); JSON is not a PubMed EFetch format
- `retmode=text&rettype=abstract` or `retmode=text&rettype=medline`: Text views
- `api_key`: Optional NCBI API key; `tool` and `email` identify the caller

**Example URL**:
```
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?
  db=pubmed&
  id=12345678,12345679&
  retmode=xml&
  api_key=YOUR_API_KEY
```

**Response**: XML with complete metadata including:
- Title
- Authors (with affiliations)
- Abstract
- Journal
- Publication date
- DOI
- PMID, PMCID
- MeSH terms
- Keywords

### ESummary - Get Summaries

Lighter-weight alternative to EFetch.

**Example**:
```
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?
  db=pubmed&
  id=12345678&
  retmode=json&
  api_key=YOUR_API_KEY
```

**Returns**: JSON `result.uids` plus records keyed by PMID; this is not ESearch's
`esearchresult` shape. This endpoint is an illustrative manual alternative; the
bundled search script uses ESearch and EFetch only.

### ELink - Find Related Articles

Find related articles or links to other databases.

**Example**:
```
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?
  dbfrom=pubmed&
  db=pubmed&
  id=12345678&
  linkname=pubmed_pubmed_citedin&
  cmd=neighbor&
  retmode=xml
```

**Link types**:
- `pubmed_pubmed`: Related articles
- `pubmed_pubmed_citedin`: Papers citing this article
- `pubmed_pmc`: PMC full-text versions (set target `db=pmc`)
- `pubmed_protein`: Related protein records

ELink XML contains `LinkSet/LinkSetDb/Link/Id` when matching links exist; an
empty link set is valid. The cited-in relationship is not a complete citation
index. EInfo (`einfo.fcgi?db=pubmed&retmode=json`) reports database fields and
link names, not article records. These are illustrative manual API calls.

### Rate Limiting

**Without API key**:
- 3 requests per second
- Block if exceeded

**With API key**:
- 10 requests per second
- Better for programmatic access

**Best practice**:
```python
import time
time.sleep(0.34)  # ~3 requests/second
# or
time.sleep(0.11)  # ~10 requests/second with API key
```

### API Key Usage

**Get API key**:
1. Create NCBI account: https://www.ncbi.nlm.nih.gov/account/
2. Settings → API Key Management
3. Create new API key
4. Copy key

**Use in requests**:
```
&api_key=YOUR_API_KEY_HERE
```

**Store securely**:
```bash
# In environment variable
export NCBI_API_KEY="your_key_here"

# In script
import os
api_key = os.getenv('NCBI_API_KEY')
```

## Search Strategies

### Comprehensive Systematic Search

For systematic reviews and meta-analyses:

```
# 1. Identify key concepts
Concept 1: Diabetes
Concept 2: Treatment
Concept 3: Outcomes

# 2. Find MeSH terms and synonyms
Concept 1: "Diabetes Mellitus"[MeSH] OR diabetes OR diabetic
Concept 2: "Drug Therapy"[MeSH] OR treatment OR therapy OR medication
Concept 3: "Treatment Outcome"[MeSH] OR outcome OR efficacy OR effectiveness

# 3. Combine with AND
("Diabetes Mellitus"[MeSH] OR diabetes OR diabetic) 
  AND ("Drug Therapy"[MeSH] OR treatment OR therapy OR medication)
  AND ("Treatment Outcome"[MeSH] OR outcome OR efficacy OR effectiveness)

# 4. Add filters
AND 2015:2024[Publication Date]
AND ("Clinical Trial"[Publication Type] OR "Randomized Controlled Trial"[Publication Type])
AND English[Language]
AND humans[MeSH Terms]
```

### Finding Clinical Trials

```
# Specific disease + clinical trials
"Alzheimer Disease"[MeSH] 
  AND ("Clinical Trial"[Publication Type] 
       OR "Randomized Controlled Trial"[Publication Type])
  AND 2020:2024[Publication Date]

# Specific drug trials
"Metformin"[MeSH] 
  AND "Diabetes Mellitus, Type 2"[MeSH]
  AND "Randomized Controlled Trial"[Publication Type]
```

### Finding Reviews

```
# Systematic reviews on topic
"CRISPR-Cas Systems"[MeSH] 
  AND ("Systematic Review"[Publication Type] OR "Meta-Analysis"[Publication Type])

# Reviews in high-impact journals
cancer immunotherapy 
  AND "Review"[Publication Type]
  AND ("Nature"[Journal] OR "Science"[Journal] OR "Cell"[Journal])
```

### Finding Recent Papers

```
# Papers from last year
"machine learning"[Title/Abstract] 
  AND "drug discovery"[Title/Abstract]
  AND 2024[Publication Date]

# Recent papers in specific journal
"CRISPR"[Title/Abstract] 
  AND "Nature"[Journal]
  AND 2023:2024[Publication Date]
```

### Author Tracking

```
# Specific author's recent work
"Doudna JA"[Author] AND 2020:2024[Publication Date]

# Author + topic
"Church GM"[Author] AND "synthetic biology"[Title/Abstract]
```

### High-Quality Evidence

```
# Meta-analyses and systematic reviews
(diabetes OR "diabetes mellitus") 
  AND (treatment OR therapy)
  AND ("Meta-Analysis"[Publication Type] OR "Systematic Review"[Publication Type])

# RCTs only
cancer immunotherapy 
  AND "Randomized Controlled Trial"[Publication Type]
  AND 2020:2024[Publication Date]
```

## Script Integration

The bundled client fetches explicit PMIDs in batches of 200, checks each batch
for completeness, and refuses to export when any batch fails or includes
unsupported PubmedBookArticle records. It does not export the full PubMed XML.

### search_pubmed.py Usage

**Basic search**:
```bash
python scripts/search_pubmed.py "diabetes treatment"
```

**With MeSH terms**:
```bash
python scripts/search_pubmed.py \
  --query '"Diabetes Mellitus"[MeSH] AND "Drug Therapy"[MeSH]'
```

**Date range filter**:
```bash
python scripts/search_pubmed.py "CRISPR" \
  --date-start 2020-01-01 \
  --date-end 2024-12-31 \
  --limit 200
```

**Publication type filter**:
```bash
python scripts/search_pubmed.py "cancer immunotherapy" \
  --publication-types "Clinical Trial,Randomized Controlled Trial" \
  --limit 100
```

**Export to BibTeX**:
```bash
python scripts/search_pubmed.py "Alzheimer's disease" \
  --limit 100 \
  --format bibtex \
  --output alzheimers.bib
```

**Complex query from file**:
```bash
# Save complex query in query.txt
cat > query.txt << 'EOF'
("Diabetes Mellitus, Type 2"[MeSH] OR "diabetes"[Title/Abstract])
AND ("Metformin"[MeSH] OR "metformin"[Title/Abstract])
AND "Randomized Controlled Trial"[Publication Type]
AND 2015:2024[Publication Date]
AND English[Language]
EOF

# Run search
python scripts/search_pubmed.py --query-file query.txt --limit 500
```

### Batch Searches

```bash
# Search multiple topics
TOPICS=("diabetes treatment" "cancer immunotherapy" "CRISPR gene editing")

for topic in "${TOPICS[@]}"; do
  python scripts/search_pubmed.py "$topic" \
    --limit 100 \
    --output "${topic// /_}.json"
  sleep 1
done
```

### Extract Metadata

```bash
# Search returns PMIDs
python scripts/search_pubmed.py "topic" --output results.json

# Extract full metadata
python scripts/extract_metadata.py \
  --input results.json \
  --output references.bib
```

## Tips and Best Practices

### Search Construction

1. **Start with MeSH terms**:
   - Use MeSH Browser to find correct terms
   - More precise than keyword search
   - Combine with text words: not every PubMed record has MeSH indexing

2. **Include text word variants**:
   ```
   # Better coverage
   ("Diabetes Mellitus"[MeSH] OR diabetes OR diabetic)
   ```

3. **Use field tags appropriately**:
   - `[MeSH]` for standardized concepts
   - `[Title/Abstract]` for specific terms
   - `[Author]` for known authors
   - `[Journal]` for specific venues

4. **Build incrementally**:
   ```
   # Step 1: Basic search
   diabetes
   
   # Step 2: Add specificity
   "Diabetes Mellitus, Type 2"[MeSH]
   
   # Step 3: Add treatment
   "Diabetes Mellitus, Type 2"[MeSH] AND "Metformin"[MeSH]
   
   # Step 4: Add study type
   "Diabetes Mellitus, Type 2"[MeSH] AND "Metformin"[MeSH] 
     AND "Clinical Trial"[Publication Type]
   
   # Step 5: Add date range
   ... AND 2020:2024[Publication Date]
   ```

### Optimizing Results

1. **Too many results**: Add filters
   - Restrict publication type
   - Narrow date range
   - Add more specific MeSH terms
   - Use Major Topic: `[MeSH Major Topic]`

2. **Too few results**: Broaden search
   - Remove restrictive filters
   - Use OR for synonyms
   - Expand date range
   - Use MeSH explosion (default)

3. **Irrelevant results**: Refine terms
   - Use more specific MeSH terms
   - Add exclusions with NOT
   - Use Title field instead of all fields
   - Add MeSH subheadings

### Quality Control

1. **Document search strategy**:
   - Save exact query string
   - Record search date
   - Note number of results
   - Save filters used

2. **Export systematically**:
   - Use consistent file naming
   - Export to JSON for flexibility
   - Convert to BibTeX as needed
   - Keep original search results

3. **Validate retrieved citations**:
   ```bash
   python scripts/validate_citations.py pubmed_results.bib
   ```

### Staying Current

1. **Set up search alerts**:
   - PubMed → Save search
   - Receive email updates
   - Daily, weekly, or monthly

2. **Track specific journals**:
   ```
   "Nature"[Journal] AND CRISPR[Title]
   ```

3. **Follow key authors**:
   ```
   "Church GM"[Author]
   ```

## Common Issues and Solutions

### Issue: MeSH Term Not Found

**Solution**: 
- Check spelling
- Use MeSH Browser
- Try related terms
- Use text word search as fallback

### Issue: Zero Results

**Solution**:
- Remove filters
- Check query syntax
- Use OR for broader search
- Try synonyms

### Issue: Poor Quality Results

**Solution**:
- Add publication type filters
- Restrict to recent years
- Use MeSH Major Topic
- Filter by journal quality

### Issue: Duplicates from Different Sources

**Solution**:
```bash
python scripts/format_bibtex.py results.bib \
  --deduplicate \
  --output clean.bib
```

### Issue: API Rate Limiting

**Solution**:
- Get API key (increases limit to 10/sec)
- Add delays in scripts
- Process in batches
- Use off-peak hours

## Summary

PubMed provides authoritative biomedical literature search:

✓ **Curated content**: MeSH indexing, quality control  
✓ **Precise search**: Field tags, MeSH terms, filters  
✓ **Programmatic access**: E-utilities API  
✓ **Free access**: No subscription required  
✓ **Comprehensive**: 35M+ citations, daily updates  

Key strategies:
- Use MeSH terms for precise searching
- Combine with text words for comprehensive coverage
- Apply appropriate field tags
- Filter by publication type and date
- Use E-utilities API for automation
- Document search strategy for reproducibility

For broader coverage across disciplines, complement with Google Scholar.

reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall