LeoYeAI/agent-master-skills / skills/biopython
biopython
VerifiedProvides Biopython workflows for sequence manipulation, file parsing (FASTA/GenBank/PDB), phylogenetics, and programmatic NCBI/PubMed access (Bio.Entrez). Supports batch processing, custom molecular-biology pipelines, BLAST automation, structure analysis, and motif analysis.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 16.8 KB |
| references/advanced.md | markdown | 15.7 KB |
| references/alignment.md | markdown | 9.7 KB |
| references/blast.md | markdown | 13.7 KB |
| references/databases.md | markdown | 13.5 KB |
| references/phylogenetics.md | markdown | 14.8 KB |
| references/review.md | markdown | 6.2 KB |
| references/sequence_io.md | markdown | 8.0 KB |
| references/structure.md | markdown | 13.1 KB |
references/databases.md
View on GitHub# Database Access with Bio.Entrez
## Overview
Bio.Entrez provides programmatic access to NCBI's Entrez databases, including PubMed, GenBank, Gene, Protein, Nucleotide, and many others. It provides request construction, process-local pacing, retries, and XML parsing. Applications must still coordinate shared IP/API-key limits across workers and validate database-specific results.
## Setup and Configuration
### Email Address (Required)
NCBI requires an email address to track usage and contact users if issues arise:
```python
from Bio import Entrez
# Always set your email
Entrez.email = "your.email@example.com"
# For reusable software, set a stable tool name and register
# the tool/email pair with NCBI.
Entrez.tool = "your_tool_name"
```
### API Key (Recommended)
Using an API key increases E-utilities rate limits from 3 to 10 requests per second. Register at https://www.ncbi.nlm.nih.gov/account/settings/ and read only `NCBI_API_KEY` from the environment — do not hardcode it and do not load unrelated environment variables:
```python
import os
# Check NCBI_API_KEY from the environment (or export it before running)
if api_key := os.environ.get("NCBI_API_KEY"):
Entrez.api_key = api_key
```
### Rate Limiting
For serial calls in one process, Biopython paces requests at the standard limits:
- **Without API key**: 3 requests per second
- **With API key**: 10 requests per second
The module handles ordinary request pacing automatically and retries transient HTTP 5XX failures. For large jobs, use Entrez history (`usehistory="y"` / `WebEnv` / `query_key`) and batch requests rather than making one request per record. NCBI also recommends scheduling large jobs for weekends or between 9:00 PM and 5:00 AM Eastern time.
## Core Entrez Functions
### EInfo - Database Information
Get information about available databases and their statistics:
```python
# List all databases
handle = Entrez.einfo()
result = Entrez.read(handle)
handle.close()
print(result["DbList"])
# Get information about a specific database
handle = Entrez.einfo(db="pubmed")
result = Entrez.read(handle)
handle.close()
print(result["DbInfo"]["Description"])
print(result["DbInfo"]["Count"]) # Number of records
```
### ESearch - Search Databases
Search for records and retrieve their IDs:
```python
# Search PubMed
handle = Entrez.esearch(db="pubmed", term="biopython")
result = Entrez.read(handle)
handle.close()
id_list = result["IdList"]
count = result["Count"]
print(f"Found {count} results")
print(f"Retrieved IDs: {id_list}")
```
### Advanced ESearch Parameters
```python
# Search with additional parameters
handle = Entrez.esearch(
db="pubmed",
term="biopython[Title]",
retmax=100, # Return up to 100 IDs
sort="relevance", # Sort by relevance
reldate=365, # Only results from last year
datetype="pdat" # Use publication date
)
result = Entrez.read(handle)
handle.close()
```
### ESummary - Get Record Summaries
Retrieve summary information for a list of IDs:
```python
# Get summaries for multiple records
handle = Entrez.esummary(db="pubmed", id="19304878,18606172")
results = Entrez.read(handle)
handle.close()
for record in results:
print(f"Title: {record['Title']}")
print(f"Authors: {record['AuthorList']}")
print(f"Journal: {record['Source']}")
print()
```
### EFetch - Retrieve Full Records
Fetch complete records in various formats:
```python
# Fetch a GenBank record
handle = Entrez.efetch(db="nucleotide", id="EU490707", rettype="gb", retmode="text")
record_text = handle.read()
handle.close()
# Parse with SeqIO
from Bio import SeqIO
handle = Entrez.efetch(db="nucleotide", id="EU490707", rettype="gb", retmode="text")
record = SeqIO.read(handle, "genbank")
handle.close()
print(record.description)
```
### EFetch Return Types
Different databases support different return types:
**Nucleotide/Protein:**
- `rettype="fasta"` - FASTA format
- `rettype="gb"` - GenBank text for nucleotide records (parse with `SeqIO`, `"genbank"`)
- `rettype="gp"` - GenPept format (proteins)
**PubMed:**
- `rettype="medline"` - MEDLINE format
- `rettype="abstract"` - Abstract text
**Common modes:**
- `retmode="text"` - Plain text
- `retmode="xml"` - XML format
### ELink - Find Related Records
Find links between records in different databases:
```python
# Find protein records linked to a nucleotide record
with Entrez.esearch(db="nuccore", term="EU490707[Accession]", retmax=1) as handle:
source_ids = Entrez.read(handle)["IdList"]
if not source_ids:
raise ValueError("Sequence accession was not found")
handle = Entrez.elink(dbfrom="nuccore", db="protein", id=source_ids)
result = Entrez.read(handle)
handle.close()
# Extract linked IDs
for linkset in result[0]["LinkSetDb"]:
if linkset["LinkName"] == "nuccore_protein":
protein_ids = [link["Id"] for link in linkset["Link"]]
print(f"Linked protein IDs: {protein_ids}")
```
### EPost - Upload ID Lists
Upload large lists of IDs to the server for later use:
```python
# Post IDs to server
id_list = ["19304878", "18606172", "16403221"]
handle = Entrez.epost(db="pubmed", id=",".join(id_list))
result = Entrez.read(handle)
handle.close()
# Get query_key and WebEnv for later use
query_key = result["QueryKey"]
webenv = result["WebEnv"]
# Use in subsequent queries
handle = Entrez.efetch(
db="pubmed",
query_key=query_key,
WebEnv=webenv,
rettype="medline",
retmode="text"
)
```
### Counts Across Selected Databases
`Entrez.egquery` was removed in Biopython 1.87 because its NCBI service stopped
working. Use explicit, serial ESearch calls for the databases actually needed:
```python
for db in ("pubmed", "nucleotide", "protein"):
with Entrez.esearch(db=db, term="biopython", retmax=0) as handle:
result = Entrez.read(handle)
print(db, int(result["Count"]))
```
### ESpell - Spelling Suggestions
Get spelling suggestions for search terms:
```python
handle = Entrez.espell(db="pubmed", term="biopythn")
result = Entrez.read(handle)
handle.close()
print(f"Original: {result['Query']}")
print(f"Suggestion: {result['CorrectedQuery']}")
```
## Working with Different Databases
### PubMed
```python
# Search for articles
handle = Entrez.esearch(db="pubmed", term="cancer genomics", retmax=10)
result = Entrez.read(handle)
handle.close()
# Fetch abstracts
handle = Entrez.efetch(
db="pubmed",
id=result["IdList"],
rettype="medline",
retmode="text"
)
records = handle.read()
handle.close()
print(records)
```
### GenBank / Nucleotide
```python
# Search for sequences
handle = Entrez.esearch(db="nucleotide", term="Cypripedioideae[Orgn] AND matK[Gene]")
result = Entrez.read(handle)
handle.close()
# Fetch sequences
if result["IdList"]:
handle = Entrez.efetch(
db="nucleotide",
id=result["IdList"][:5],
rettype="fasta",
retmode="text"
)
sequences = handle.read()
handle.close()
```
### Protein
```python
# Search for protein sequences
handle = Entrez.esearch(db="protein", term="human insulin")
result = Entrez.read(handle)
handle.close()
# Fetch protein records
from Bio import SeqIO
handle = Entrez.efetch(
db="protein",
id=result["IdList"][:5],
rettype="gp",
retmode="text"
)
records = SeqIO.parse(handle, "genbank")
for record in records:
print(f"{record.id}: {record.description}")
handle.close()
```
### Gene
```python
# Search for gene records
handle = Entrez.esearch(db="gene", term="BRCA1[Gene] AND human[Organism]")
result = Entrez.read(handle)
handle.close()
# Get gene information
handle = Entrez.efetch(db="gene", id=result["IdList"][0], retmode="xml")
record = Entrez.read(handle)
handle.close()
```
### Taxonomy
```python
# Search for organism
handle = Entrez.esearch(db="taxonomy", term="Homo sapiens")
result = Entrez.read(handle)
handle.close()
# Fetch taxonomic information
handle = Entrez.efetch(db="taxonomy", id=result["IdList"][0], retmode="xml")
records = Entrez.read(handle)
handle.close()
for record in records:
print(f"TaxID: {record['TaxId']}")
print(f"Scientific Name: {record['ScientificName']}")
print(f"Lineage: {record['Lineage']}")
```
## Parsing Entrez Results
### Reading XML Results
Use `Entrez.read` for supported NCBI XML, not JSON or text. Saved XML must be
opened with `"rb"`; text formats use their own parser. Check for empty ID lists
before EFetch or indexing `[0]`, and record `QueryTranslation` and retrieval date.
```python
# Most results can be parsed with Entrez.read()
handle = Entrez.efetch(db="pubmed", id="19304878", retmode="xml")
records = Entrez.read(handle)
handle.close()
# Access parsed data
article = records['PubmedArticle'][0]['MedlineCitation']['Article']
print(article['ArticleTitle'])
```
### Handling Large Result Sets
PubMed ESearch can retrieve only the first 10,000 matches; `retstart` and history
do not bypass that cap. Partition by dates or other reproducible filters, verify
each partition count, and deduplicate PMIDs, or use NCBI EDirect/bulk data.
```python
# Batch processing for large searches
search_term = "cancer[Title]"
handle = Entrez.esearch(db="pubmed", term=search_term, retmax=0)
result = Entrez.read(handle)
handle.close()
total_count = int(result["Count"])
if total_count > 10_000:
raise ValueError("Partition this PubMed query into verified <=10,000-result queries")
batch_size = 500
for start in range(0, total_count, batch_size):
# Fetch batch
handle = Entrez.esearch(
db="pubmed",
term=search_term,
retstart=start,
retmax=batch_size
)
result = Entrez.read(handle)
handle.close()
# Process IDs
id_list = result["IdList"]
print(f"Processing IDs {start} to {start + len(id_list)}")
```
## Advanced Patterns
### Search History with WebEnv
```python
# Perform search and store on server
handle = Entrez.esearch(
db="pubmed",
term="biopython",
usehistory="y"
)
result = Entrez.read(handle)
handle.close()
webenv = result["WebEnv"]
query_key = result["QueryKey"]
count = int(result["Count"])
if count > 10_000:
raise ValueError("Partition the PubMed query before fetching from history")
# Fetch results in batches using history
batch_size = 100
for start in range(0, count, batch_size):
handle = Entrez.efetch(
db="pubmed",
retstart=start,
retmax=batch_size,
rettype="medline",
retmode="text",
WebEnv=webenv,
query_key=query_key
)
data = handle.read()
handle.close()
# Process data
```
### Combining Searches
```python
# Use boolean operators
complex_search = "(cancer[Title]) AND (genomics[Title]) AND 2020:2025[PDAT]"
handle = Entrez.esearch(db="pubmed", term=complex_search, retmax=100)
result = Entrez.read(handle)
handle.close()
```
## Best Practices
1. **Always set Entrez.email** - Required by NCBI
2. **Set Entrez.tool** for reusable software and register the tool/email pair with NCBI
3. **Use `NCBI_API_KEY` from the environment** for higher rate limits (10 req/s vs 3 req/s); never hardcode keys
4. **Close handles** after reading to free resources
5. **Batch large requests** - Use retstart and retmax within the database cap
6. **Use WebEnv for bounded downloads** - Store results on server; do not log temporary history tokens or API-key-bearing handle URLs
7. **Cache locally** - Download once and save to avoid repeated requests
8. **Handle errors gracefully** - Network issues and API limits can occur
9. **Respect NCBI guidelines** - Don't overwhelm the service
10. **Use appropriate rettype** - Choose format that matches your needs
11. **Parse XML carefully** - Structure varies by database and record type
## Error Handling
```python
from urllib.error import HTTPError
from Bio import Entrez
Entrez.email = "your.email@example.com"
try:
handle = Entrez.efetch(db="nucleotide", id="invalid_id", rettype="gb")
record = handle.read()
handle.close()
except HTTPError as e:
print(f"HTTP Error: {e.code} - {e.reason}")
except Exception as e:
print(f"Error: {e}")
```
## Common Use Cases
### Download GenBank Records
```python
from Bio import Entrez, SeqIO
Entrez.email = "your.email@example.com"
# List of accession numbers
accessions = ["EU490707", "EU490708", "EU490709"]
for acc in accessions:
handle = Entrez.efetch(db="nucleotide", id=acc, rettype="gb", retmode="text")
record = SeqIO.read(handle, "genbank")
handle.close()
# Save to file
SeqIO.write(record, f"{acc}.gb", "genbank")
```
### Search and Download Papers
```python
# Search PubMed
handle = Entrez.esearch(db="pubmed", term="machine learning bioinformatics", retmax=20)
result = Entrez.read(handle)
handle.close()
# Get details
handle = Entrez.efetch(db="pubmed", id=result["IdList"], retmode="xml")
papers = Entrez.read(handle)
handle.close()
# Extract information
for paper in papers['PubmedArticle']:
article = paper['MedlineCitation']['Article']
print(f"Title: {article['ArticleTitle']}")
print(f"Journal: {article['Journal']['Title']}")
print()
```
### Find Related Sequences
```python
# Start with one sequence
handle = Entrez.efetch(db="nucleotide", id="EU490707", rettype="gb", retmode="text")
record = SeqIO.read(handle, "genbank")
handle.close()
# Find similar sequences
with Entrez.esearch(db="nuccore", term="EU490707[Accession]", retmax=1) as handle:
source_ids = Entrez.read(handle)["IdList"]
if not source_ids:
raise ValueError("Sequence accession was not found")
handle = Entrez.elink(dbfrom="nuccore", db="nuccore", id=source_ids)
result = Entrez.read(handle)
handle.close()
# Get related IDs
related_ids = []
for linkset in result[0]["LinkSetDb"]:
for link in linkset["Link"]:
related_ids.append(link["Id"])
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |