DAAF-Contribution-Community/daaf / .claude/skills/education-data-source-nhgis
education-data-source-nhgis
VerifiedNHGIS — census geography crosswalks via Portal: links schools (ncessch) and colleges (unitid) to census tracts, block groups, CBSAs, and regions (census-geography vintages 1990-2020; mirror file-year/linkage coverage 1980, 1984-2023). Portal provides geography linkage tables ONLY; browse census demographics in the public NHGIS Data Finder, then sign in with a free IPUMS NHGIS account to submit, manage, or download extracts. API extract requests also need an API key. Use for linking education or institutional data to census geography.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 23.6 KB |
| references/boundary-changes.md | markdown | 9.1 KB |
| references/data-access.md | markdown | 17.0 KB |
| references/geographic-units.md | markdown | 8.6 KB |
| references/school-geography-links.md | markdown | 14.5 KB |
| references/time-series.md | markdown | 10.9 KB |
| references/variable-catalog.md | markdown | 18.4 KB |
references/data-access.md
View on GitHub# Accessing NHGIS Data
Methods for obtaining NHGIS data: via Education Data Portal mirrors (school/college-to-census links) and via direct NHGIS access (custom census analysis).
## Quick Start: Education Data Portal (School/College-Census Links)
For linking schools or colleges to census geography, use the pre-processed data from the Education Data Portal mirrors. **No registration required.**
```python
import polars as pl
# Uses fetch_from_mirrors() — tries each mirror in priority order per mirrors.yaml.
# See fetch-patterns.md and datasets-reference.md for canonical paths.
# School-to-census geography (2020 Census boundaries)
df = fetch_from_mirrors("nhgis/schools_nhgis_geog_2020")
# College-to-census geography (2020 Census boundaries)
df_colleges = fetch_from_mirrors("nhgis/colleges_nhgis_geog_2020")
# Available census years: 1990, 2000, 2010, 2020
# All files contain ALL data years (schools: 1986-2023, colleges: 1980, 1984-2023)
# The census year determines which boundary vintage is used for geographic assignment
```
```r
library(arrow)
# fetch_from_mirrors() is a Python helper; in R, build URLs from the mirror root.
# Mirror failover: see `education-data-query/references/fetch-patterns.md` (R pattern);
# canonical paths: datasets-reference.md.
mirror <- yaml::read_yaml("mirrors.yaml")$mirrors[[1]]
# School-to-census geography (2020 Census boundaries)
# NOTE: illustrative only — mirror parquet files are Polars-written and may
# declare string_view columns, so a plain read can fail under R arrow
# ("cannot handle Array of type <utf8_view>"). Real fetch scripts must use the
# view-safe parquet read from `education-data-query/references/fetch-patterns.md`.
df <- read_parquet(paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "nhgis/schools_nhgis_geog_2020", ".", mirror$format))
# College-to-census geography (2020 Census boundaries)
df_colleges <- read_parquet(paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "nhgis/colleges_nhgis_geog_2020", ".", mirror$format))
# Available census years: 1990, 2000, 2010, 2020
# All files contain ALL data years (schools: 1986-2023, colleges: 1980, 1984-2023)
# The census year determines which boundary vintage is used for geographic assignment
```
**Available data**: Institution coordinates linked to census tract, block group, block, region, division, CBSA, and place. See `variable-catalog.md` for Portal integer encodings.
Codebooks are `.xls` files co-located with data in all mirrors. Use `get_codebook_url()` from `fetch-patterns.md` to construct download URLs. Codebook paths are listed in `datasets-reference.md`.
> **Truth Hierarchy:** When interpreting variable values, apply this priority:
> 1. **Actual data file** (what you observe in the parquet/CSV) — this IS the truth
> 2. **Live codebook** (.xls in mirror) — authoritative documentation, may lag
> 3. **This skill documentation** — convenient summary, may drift from codebook
>
> If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate.
**For custom census analysis** (tract-level demographics, time series, boundary files): Use NHGIS directly via methods below.
---
## Direct NHGIS Access (Custom Analysis)
For full census data access beyond school/college-census links, use NHGIS directly. This is separate from the Education Data Portal mirror system.
### Browsing, Account, and API Requirements
The NHGIS Data Finder at `https://data2.nhgis.org/main` can be browsed without signing in: users can inspect datasets, tables, years, topics, and geographic levels publicly. A free IPUMS NHGIS account and login are required to **submit, manage, and download extracts**. Register at `https://uma.pop.umn.edu/nhgis/user/new`.
API extract requests have an additional requirement: an IPUMS API key associated with the registered account. See `https://developer.ipums.org/docs/v2/get-started/` for current prerequisites.
Authentication therefore applies to extract operations, not to ordinary Data Finder browsing. Account access provides:
- Extract submission, history, management, and downloads
- API-key creation for programmatic extract requests
## Method 1: NHGIS Data Finder (Web Interface)
### URL
https://data2.nhgis.org/main
### Workflow
1. **Filter data** using sidebar options:
- Geographic Levels (State, County, Tract, etc.)
- Years (1790-2024 as of the February 2026 ACS release; check the live filter)
- Topics (Population, Housing, etc.)
- Datasets (Decennial, ACS periods)
2. **Select data type** (tabs):
- SOURCE TABLES: Original census tables
- TIME SERIES TABLES: Harmonized across years
- GIS FILES: Boundary shapefiles
3. **Add to Data Cart**:
- Click "+" to add tables/files
- Review cart contents
4. **Configure extract** (Continue button):
- Select geographic levels for tables
- Choose output format (CSV; optional SAS/SPSS/Stata control files)
- Select layout for time series
5. **Submit extract**:
- Extracts queued on server
- Email notification when ready
- Download from Extracts History page
### Output Formats
| Format | Extension | Best For |
|--------|-----------|----------|
| CSV | .csv | General use, Excel, Python, R |
| Shapefile | .shp | GIS software |
> NHGIS discontinued fixed-width table extracts on 2026-02-11; table data are delivered as CSV only, with optional control files for SAS, SPSS, or Stata (NHGIS Revision History).
### Extract Contents
Each extract includes:
- Data file(s) (.csv)
- Codebook (.txt) - variable descriptions
- For GIS: multiple shapefile components
## Method 2: IPUMS API
### Overview
RESTful API for programmatic data requests.
**Documentation**: https://developer.ipums.org/docs/v2/apiprogram/
### Authentication
```bash
# API key in header
curl -X GET "https://api.ipums.org/metadata/datasets?collection=nhgis&version=2" \
-H "Authorization: YOUR_API_KEY"
```
### Endpoints
| Endpoint | Purpose |
|----------|---------|
| `/metadata/datasets?collection=nhgis&version=2` | List available datasets |
| `/metadata/data_tables?collection=nhgis&version=2` | List tables |
| `/metadata/time_series_tables?collection=nhgis&version=2` | List time series |
| `/extracts/` | Create and manage extracts |
### Creating Extract via API
```python
import requests
api_key = "YOUR_API_KEY"
headers = {"Authorization": api_key}
# Define extract
extract_def = {
"datasets": {
"2016_2020_ACS5a": {
"dataTables": ["B01001"],
"geogLevels": ["tract"]
}
},
"dataFormat": "csv_no_header",
"description": "Population by age, tract level"
}
# Submit extract (IPUMS API Version 2)
response = requests.post(
"https://api.ipums.org/extracts/?collection=nhgis&version=2",
headers=headers,
json=extract_def
)
extract_id = response.json()["number"]
```
```r
library(httr2)
api_key <- "YOUR_API_KEY"
# Define extract
extract_def <- list(
datasets = list(
`2016_2020_ACS5a` = list(
dataTables = list("B01001"),
geogLevels = list("tract")
)
),
dataFormat = "csv_no_header",
description = "Population by age, tract level"
)
# Submit extract
resp <- request("https://api.ipums.org/extracts/?collection=nhgis&version=2") |>
req_headers(Authorization = api_key) |>
req_body_json(extract_def) |>
req_perform()
extract_id <- resp_body_json(resp)$number
```
### Checking Extract Status
```python
status_url = f"https://api.ipums.org/extracts/{extract_id}?collection=nhgis&version=2"
response = requests.get(status_url, headers=headers)
status = response.json()["status"]
# "queued", "started", "produced", "canceled", "failed", "completed"
```
```r
status_url <- paste0(
"https://api.ipums.org/extracts/", extract_id,
"?collection=nhgis&version=2"
)
response <- request(status_url) |> req_headers(Authorization = api_key) |> req_perform()
status <- resp_body_json(response)$status
# "queued", "started", "produced", "canceled", "failed", "completed"
```
### Downloading Extract
```python
if status == "completed":
download_links = response.json()["downloadLinks"]
for name, link in download_links.items():
data = requests.get(link["url"], headers=headers)
# Save to file
```
```r
if (status == "completed") {
download_links <- resp_body_json(response)$downloadLinks
for (link in download_links) {
download.file(link$url, destfile = basename(link$url),
headers = c(Authorization = api_key))
}
}
```
## Method 3: ipumspy (Python)
### Availability
`ipumspy` is **not** pre-installed in DAAF, and runtime installs are blocked
(`pip install` is refused at the command line and inside executed scripts — see
CLAUDE.md § Runtime Package Installation). This access method is therefore
unavailable until `ipumspy` is added to the Dockerfile (user additions block near
the end) and the container is rebuilt. The direct-API methods shown above need no
extra package and work today.
### Configuration
```python
from ipumspy import IpumsApiClient, readers
# Initialize client
client = IpumsApiClient(api_key="YOUR_API_KEY")
```
### Creating and Submitting Extract
```python
from ipumspy.nhgis import NhgisExtract
# Define extract
extract = NhgisExtract(
datasets={
"2016_2020_ACS5a": {
"data_tables": ["B01001", "B19013"],
"geog_levels": ["tract"]
}
}
)
# Submit
client.submit_extract(extract)
print(f"Extract number: {extract.extract_id}")
```
### Waiting and Downloading
```python
# Wait for completion
client.wait_for_extract(extract)
# Download
client.download_extract(extract, download_dir="./data")
```
### Reading NHGIS Data
```python
from ipumspy import readers
# Read tabular data with codebook
ddi = readers.read_nhgis_codebook("nhgis_extract_codebook.txt")
data = readers.read_nhgis("nhgis_data.csv", ddi)
```
### Example: Complete Workflow
```python
import polars as pl
from ipumspy import IpumsApiClient
from ipumspy.nhgis import NhgisExtract
# Setup
client = IpumsApiClient("YOUR_API_KEY")
# Define extract: Poverty data for tracts
extract = NhgisExtract(
datasets={
"2018_2022_ACS5a": {
"data_tables": ["B17001"], # Poverty status
"geog_levels": ["tract"]
}
},
geographic_extents=["California"], # Single state
data_format="csv_header"
)
# Submit and wait
client.submit_extract(extract)
client.wait_for_extract(extract)
# Download
files = client.download_extract(extract, download_dir="./nhgis_data")
# Read data with Polars
df = pl.read_csv(files["data"][0])
```
## Method 4: ipumsr (R)
### Availability
`ipumsr` is **not** pre-installed in DAAF, and runtime installs are blocked
(`install.packages()` is refused at the command line and inside executed scripts —
see CLAUDE.md § Runtime Package Installation). This access method is therefore
unavailable until `ipumsr` is added to the Dockerfile (user additions block near
the end) and the container is rebuilt. The direct-API methods shown above need no
extra package and work today.
### Configuration
```r
library(ipumsr)
# Set API key (or use environment variable IPUMS_API_KEY)
set_ipums_api_key("YOUR_API_KEY")
```
### Creating Extract
```r
# Define NHGIS extract
extract <- define_extract_nhgis(
description = "Tract poverty data",
datasets = ds_spec(
"2018_2022_ACS5a",
data_tables = "B17001",
geog_levels = "tract"
)
)
```
### Submitting and Downloading
```r
# Submit
submitted <- submit_extract(extract)
# Wait for completion
completed <- wait_for_extract(submitted)
# Download
downloaded <- download_extract(completed, download_dir = "./data")
```
### Reading Data
```r
# Read with variable labels
nhgis_data <- read_nhgis(downloaded)
# Access codebook info
nhgis_cb <- read_nhgis_codebook(downloaded)
```
### Time Series Example
```r
# Time series extract
ts_extract <- define_extract_nhgis(
description = "Population time series",
time_series_tables = tst_spec(
"CL8", # Total population table
geog_levels = "county",
years = c("1990", "2000", "2010", "2020")
)
)
submitted <- submit_extract(ts_extract)
wait_for_extract(submitted)
downloaded <- download_extract(submitted)
```
## Accessing Geographic Crosswalks
Crosswalk files require a signed-in IPUMS account: `secure-assets.ipums.org` crosswalk links redirect to the IPUMS login page (observed 2026-10-19). They are also available through the IPUMS API (Version 2 supplemental-data endpoints, added October 2024): replace `https://secure-assets.ipums.org/` with `https://api.ipums.org/supplemental-data/` and send the API key in the `Authorization` header.
### Web Download (login required)
https://www.nhgis.org/geographic-crosswalks#download
### Direct URLs (Examples; login or API key required)
```
# 2000 blocks to 2010 tracts
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_blk2000_tr2010.zip
# 2010 tracts to 2020 tracts
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_tr2010_tr2020.zip
# State-specific (faster download)
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_blk2000_blk2010_state/nhgis_blk2000_blk2010_06.zip # California
```
## Accessing GIS Files
### Via Data Finder
1. Click "GIS FILES" tab
2. Filter by geographic level and year
3. Add to cart and submit extract
### Via API/ipumspy
```python
# Include shapefiles in extract
extract = NhgisExtract(
datasets={"2016_2020_ACS5a": {...}},
shapefiles=["2020_tract"]
)
```
```r
# Include shapefiles in extract (ipumsr)
extract <- define_extract_nhgis(
description = "Data with shapefiles",
datasets = ds_spec("2016_2020_ACS5a",
data_tables = "B01001",
geog_levels = "tract"
),
shapefiles = "2020_tract"
)
```
### Direct TIGER/Line (Alternative)
For current boundaries only:
https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html
**Note**: NHGIS shapefiles are processed (clipped to coastlines, consistent projection).
## Data Format Details
### CSV Output
```csv
GISJOIN,YEAR,STATE,STATEA,COUNTY,COUNTYA,TRACTA,AJWME001,AJWME002
G0600010000100,2020,California,06,Alameda,001,000100,3245,1587
```
### Key Fields
| Field | Description |
|-------|-------------|
| GISJOIN | NHGIS geographic identifier |
| YEAR | Data year |
| STATE | State name |
| STATEA | State FIPS (with NHGIS suffix) |
| COUNTY | County name |
| COUNTYA | County FIPS (with NHGIS suffix) |
| [Variable codes] | Data values |
### Codebook Format
```
Context Fields
GISJOIN: GIS Join Match Code
YEAR: Data File Year
...
Table 1: B01001. Sex by Age
Universe: Total population
AJWME001: Total
AJWME002: Male
AJWME003: Male: Under 5 years
...
```
## Performance Tips
### Large Extracts
- Select specific states (geographic extent) when possible
- Request only needed geographic levels
- Request only the tables and years you need (fixed-width output was discontinued 2026-02-11)
### Efficient API Usage
```python
# Check if extract exists before resubmitting
existing = client.get_extract_by_id(extract_id, collection="nhgis")
if existing["status"] == "completed":
# Download existing
pass
else:
# Submit new
pass
```
```r
# Check if extract exists before resubmitting (ipumsr)
existing <- get_extract_info(submitted)
if (existing$status == "completed") {
# Download existing
} else {
# Submit new
}
```
### Batch Processing
```python
# Submit multiple extracts
extracts = []
for state in ["California", "Texas", "New York"]:
ext = NhgisExtract(
datasets={"2016_2020_ACS5a": {...}},
geographic_extents=[state]
)
client.submit_extract(ext)
extracts.append(ext)
# Wait for all
for ext in extracts:
client.wait_for_extract(ext)
```
```r
# Submit multiple extracts (ipumsr)
states <- c("California", "Texas", "New York")
extracts <- lapply(states, function(st) {
ext <- define_extract_nhgis(
description = paste("Data for", st),
datasets = ds_spec(
"2016_2020_ACS5a",
data_tables = "B01001",
geog_levels = "tract"
),
geographic_extents = st
)
submit_extract(ext)
})
# Wait for all
completed <- lapply(extracts, wait_for_extract)
```
## Troubleshooting
| Issue | Cause | Solution |
|-------|-------|----------|
| Extract stuck "queued" | Server load | Wait; check status |
| "Unauthorized" error | Invalid API key | Regenerate key in account settings |
| Empty data file | Invalid table/level combo | Check Data Finder for valid combinations |
| Missing variables | Table not available for level | Choose different geographic level |
| Shapefile mismatch | Year mismatch | Ensure shapefile year matches data year |
## Terms, Citation, and Redistribution
Direct NHGIS extracts are governed by IPUMS terms, citation requirements, and redistribution restrictions that are separate from the Education Data Portal's ODC-By terms. Review the current requirements before sharing an extract or derivative:
- Citation and use: `https://www.nhgis.org/citation-and-use-nhgis-data`
- IPUMS terms: `https://www.ipums.org/about/terms`
As verified on 2026-07-21, the citation page identifies **IPUMS NHGIS Version 21.0 (2026)**. Use the exact citation generated for the extract or shown on the current citation page rather than copying a frozen version string from this reference.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |