K-Dense-AI/scientific-agent-skills / skills/imaging-data-commons
imaging-data-commons
VerifiedQueries and downloads public cancer imaging data from NCI Imaging Data Commons. Supports IDC collection discovery, DICOM access, radiology (CT, MR, PET) and pathology AI datasets, metadata SQL, visualization, licensing, and citations. Uses public metadata and download routes without authentication; optional BigQuery and Google Healthcare routes require Google credentials.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 30.7 KB |
| references/bigquery_guide.md | markdown | 35.4 KB |
| references/cli_guide.md | markdown | 9.3 KB |
| references/clinical_data_guide.md | markdown | 12.1 KB |
| references/cloud_storage_guide.md | markdown | 14.2 KB |
| references/dicomweb_guide.md | markdown | 16.1 KB |
| references/digital_pathology_guide.md | markdown | 15.3 KB |
| references/index_tables_guide.md | markdown | 12.1 KB |
| references/licensing_and_citation.md | markdown | 10.0 KB |
| references/mcp_guide.md | markdown | 9.4 KB |
| references/parquet_access_guide.md | markdown | 9.8 KB |
| references/rest_api_guide.md | markdown | 33.3 KB |
| references/sql_patterns.md | markdown | 16.9 KB |
| references/use_cases.md | markdown | 8.5 KB |
| scripts/check_version.py | script | 5.5 KB |
references/parquet_access_guide.md
View on GitHub# Direct Parquet Access Guide for IDC
**Reviewed 2026-09-30:** idc-index-data 24.2.2 (IDC data version v24), DuckDB 1.5
All idc-index metadata tables are published as Parquet files to a public GCS bucket with unrestricted CORS access. This enables metadata queries with DuckDB or pandas without installing idc-index.
**Limitation:** download helpers (`download_from_selection()`), viewer URLs (`get_viewer_URL()`), and citation generation require the idc-index client and are not available from raw Parquet files.
**This is not the first no-install option to reach for.** It still needs DuckDB installed. This GCS artifact set contains only the `clinical_index` dictionary; full clinical and BigQuery tables are exported separately to public S3 (below). For ad-hoc metadata with nothing installed, the REST API (`rest_api_guide.md`) needs no install at all and reaches `clinical.<table>` through `POST /sql`.
## When to Use This Guide
Load this guide when you need to:
- Pin queries to a specific IDC data version (see *Pinning to a Specific Version* below) rather than whatever the hosted API currently serves
- Return more rows than the REST `/sql` ceiling of 10 000
- Run heavy or repeated local DuckDB analysis without driving the hosted API
- Query IDC metadata where DuckDB is available but idc-index is not
For downloads, viewer URLs, and citations, use idc-index as documented in the main SKILL.md.
## URL Pattern
```
https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts/{filename}.parquet
```
`current/` always resolves to the latest data release. To pin to a specific version, replace `current` with the data version number (e.g., `23.10.1`).
## Available Files
| File | Approximate Size | Description |
|------|-----------------|-------------|
| `idc_index.parquet` | ~70 MB | Primary index (all DICOM series metadata) |
| `volume_geometry_index.parquet` | ~5 MB | 3D geometry validation for CT/MR/PT series |
| `rtstruct_index.parquet` | ~2 MB | RT Structure Set ROI metadata |
| `seg_index.parquet` | ~6 MB | DICOM Segmentation cross-references |
| `sm_index.parquet` | ~2 MB | Slide microscopy series metadata |
| `contrast_index.parquet` | ~1 MB | Contrast agent metadata |
| `ann_index.parquet` | ~0.2 MB | Microscopy annotation series metadata |
| `ann_group_index.parquet` | ~0.5 MB | Annotation group metadata |
| `collections_index.parquet` | — | Collection-level metadata |
| `analysis_results_index.parquet` | — | Derived dataset metadata |
| `clinical_index.parquet` | ~0.2 MB | Clinical data column dictionary |
| `ct_index.parquet` | — | CT acquisition/reconstruction parameters |
| `mr_index.parquet` | — | MR sequence/acquisition parameters |
| `pt_index.parquet` | — | PET acquisition/radiopharmaceutical parameters |
| `prior_versions_index.parquet` | — | Series from previous IDC releases |
| `sm_instance_index.parquet` | — | Slide microscopy instance metadata |
| `version_metadata_index.parquet` | — | IDC release timestamps |
**Note:** the main index file is named `idc_index.parquet`, not `index.parquet`. Reference it with an alias in SQL queries (e.g., `FROM read_parquet(...) AS index`).
## Prerequisites
Install the Python `duckdb` package, using whatever installer manages the environment you are
running in.
DuckDB reads Parquet directly from HTTPS URLs using HTTP range requests — no GCS client library or authentication required.
## Basic Queries
```python
import duckdb
BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"
# Discover modalities and series counts
duckdb.sql(f"""
SELECT Modality, COUNT(*) as series_count, ROUND(SUM(series_size_MB)/1000, 1) as size_GB
FROM read_parquet('{BASE}/idc_index.parquet')
GROUP BY Modality
ORDER BY series_count DESC
""").df()
# Collections with CT data, ordered by size
duckdb.sql(f"""
SELECT collection_id,
COUNT(DISTINCT PatientID) as patients,
COUNT(*) as series,
ROUND(SUM(series_size_MB)/1000, 1) as size_GB
FROM read_parquet('{BASE}/idc_index.parquet')
WHERE Modality = 'CT'
GROUP BY collection_id
ORDER BY size_GB DESC
LIMIT 10
""").df()
```
## Volume Geometry Validation
`volume_geometry_index` covers single-frame CT, MR, and PT series. Each row has boolean checks for orientation, spacing, dimensions, and slice positions, plus a composite `regularly_spaced_3d_volume` flag.
```python
import duckdb
BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"
# Candidate regularly spaced CT series; verify task-specific geometry before analysis
duckdb.sql(f"""
SELECT i.collection_id, i.SeriesInstanceUID, i.BodyPartExamined,
v.obliquity_degrees, v.regularly_spaced_3d_volume
FROM read_parquet('{BASE}/idc_index.parquet') i
JOIN read_parquet('{BASE}/volume_geometry_index.parquet') v
ON i.SeriesInstanceUID = v.SeriesInstanceUID
WHERE i.Modality = 'CT'
AND v.regularly_spaced_3d_volume = TRUE
LIMIT 10
""").df()
# Fraction of 3D-valid series per collection and modality
duckdb.sql(f"""
SELECT i.collection_id, i.Modality,
COUNT(*) as total,
SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) as valid_3d,
ROUND(100.0 * SUM(CASE WHEN v.regularly_spaced_3d_volume THEN 1 ELSE 0 END) / COUNT(*), 1) as pct_valid
FROM read_parquet('{BASE}/idc_index.parquet') i
JOIN read_parquet('{BASE}/volume_geometry_index.parquet') v
ON i.SeriesInstanceUID = v.SeriesInstanceUID
WHERE i.Modality IN ('CT', 'MR', 'PT')
GROUP BY i.collection_id, i.Modality
ORDER BY total DESC
LIMIT 10
""").df()
```
Key columns in `volume_geometry_index`:
| Column | Type | Description |
|--------|------|-------------|
| `SeriesInstanceUID` | STRING | Join key |
| `single_orientation` | BOOLEAN | All instances share the same ImageOrientationPatient |
| `orthogonal_orientation` | BOOLEAN | Orientation direction cosines are orthogonal |
| `unique_slice_positions` | BOOLEAN | No duplicate or overlapping slices |
| `consistent_pixel_spacing` | BOOLEAN | All instances share the same PixelSpacing |
| `consistent_image_dimensions` | BOOLEAN | All instances share the same Rows and Columns |
| `uniform_slice_spacing` | BOOLEAN | Spacing between consecutive slices is constant |
| `obliquity_degrees` | FLOAT | Angle between slice normal and nearest cardinal axis (0 = pure axial/sagittal/coronal) |
| `regularly_spaced_3d_volume` | BOOLEAN | Composite: TRUE if all checks pass |
## RT Structure Sets
`rtstruct_index` has one row per RTSTRUCT series with aggregated ROI metadata.
```python
import duckdb
BASE = "https://storage.googleapis.com/idc-index-data-artifacts/current/release_artifacts"
# RTSTRUCT series with ROI details
duckdb.sql(f"""
SELECT i.collection_id, i.SeriesInstanceUID,
r.total_rois, r.ROINames, r.RTROIInterpretedTypes,
r.referenced_SeriesInstanceUID
FROM read_parquet('{BASE}/idc_index.parquet') i
JOIN read_parquet('{BASE}/rtstruct_index.parquet') r
ON i.SeriesInstanceUID = r.SeriesInstanceUID
WHERE i.Modality = 'RTSTRUCT'
LIMIT 5
""").df()
# Collections with the most RTSTRUCT series
duckdb.sql(f"""
SELECT i.collection_id,
COUNT(*) as rtstruct_series,
ROUND(AVG(r.total_rois), 1) as avg_rois_per_struct
FROM read_parquet('{BASE}/idc_index.parquet') i
JOIN read_parquet('{BASE}/rtstruct_index.parquet') r
ON i.SeriesInstanceUID = r.SeriesInstanceUID
GROUP BY i.collection_id
ORDER BY rtstruct_series DESC
LIMIT 10
""").df()
```
Key columns in `rtstruct_index`:
| Column | Type | Description |
|--------|------|-------------|
| `SeriesInstanceUID` | STRING | Join key (the RTSTRUCT series) |
| `total_rois` | INTEGER | Number of ROIs in the structure set |
| `ROINames` | STRING (array) | Distinct ROI names (e.g., `["GTV", "Heart", "PTV"]`) |
| `ROIGenerationAlgorithms` | STRING (array) | Distinct generation algorithms (e.g., `["AUTOMATIC", "MANUAL"]`) |
| `RTROIInterpretedTypes` | STRING (array) | Distinct ROI types (e.g., `["GTV", "ORGAN", "PTV"]`) |
| `referenced_SeriesInstanceUID` | STRING | SeriesInstanceUID of the referenced source image series |
## Pinning to a Specific Version
```python
import duckdb
# Use a specific data release instead of 'current'
VERSION = "24.2.2"
BASE = f"https://storage.googleapis.com/idc-index-data-artifacts/{VERSION}/release_artifacts"
duckdb.sql(f"SELECT COUNT(*) FROM read_parquet('{BASE}/idc_index.parquet')").df()
```
## Full metadata and clinical exports
The separate public bucket `idc-open-metadata` exports BigQuery tables, including per-
collection clinical rows, per-segment metadata, and SR measurements. No Google account
is required. Use a pinned dataset such as `idc_v24_clinical`; discover object keys first
because a table can be one file or a directory of shards. Do not append `.parquet` blindly.
```bash
s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/idc_v24_clinical/
s5cmd --no-sign-request ls s3://idc-open-metadata/bigquery_export/idc_v24/
```
For example, after confirming the key, the clinical table can be queried directly:
```python
import duckdb
url = "https://idc-open-metadata.s3.amazonaws.com/bigquery_export/idc_v24_clinical/nlst_canc/000000000000.parquet"
clinical = duckdb.sql(f"SELECT dicom_patient_id, clinical_stag FROM read_parquet('{url}') LIMIT 5").df()
```
Full instance tables can be very large. Inspect their schema and estimate resources before
querying all shards; GoogleSQL from the BigQuery guide needs adaptation to DuckDB syntax.
See [IDC files and metadata](https://learn.canceridc.dev/data/organization-of-data/files-and-metadata).
## Resources
- idc-index-data releases: https://github.com/ImagingDataCommons/idc-index-data/releases
- idc-index documentation: https://idc-index.readthedocs.io/
- IDC Portal: https://portal.imaging.datacommons.cancer.gov/
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |