Skip to content

DAAF-Contribution-Community/daaf / .claude/skills/education-data-source-nhgis

education-data-source-nhgis

Verified

NHGIS — census geography crosswalks via Portal: links schools (ncessch) and colleges (unitid) to census tracts, block groups, CBSAs, and regions (census-geography vintages 1990-2020; mirror file-year/linkage coverage 1980, 1984-2023). Portal provides geography linkage tables ONLY; browse census demographics in the public NHGIS Data Finder, then sign in with a free IPUMS NHGIS account to submit, manage, or download extracts. API extract requests also need an API key. Use for linking education or institutional data to census geography.

references/data-access.md

View on GitHub
# Accessing NHGIS Data

Methods for obtaining NHGIS data: via Education Data Portal mirrors (school/college-to-census links) and via direct NHGIS access (custom census analysis).

## Quick Start: Education Data Portal (School/College-Census Links)

For linking schools or colleges to census geography, use the pre-processed data from the Education Data Portal mirrors. **No registration required.**

```python
import polars as pl

# Uses fetch_from_mirrors() — tries each mirror in priority order per mirrors.yaml.
# See fetch-patterns.md and datasets-reference.md for canonical paths.

# School-to-census geography (2020 Census boundaries)
df = fetch_from_mirrors("nhgis/schools_nhgis_geog_2020")

# College-to-census geography (2020 Census boundaries)
df_colleges = fetch_from_mirrors("nhgis/colleges_nhgis_geog_2020")

# Available census years: 1990, 2000, 2010, 2020
# All files contain ALL data years (schools: 1986-2023, colleges: 1980, 1984-2023)
# The census year determines which boundary vintage is used for geographic assignment
```

```r
library(arrow)

# fetch_from_mirrors() is a Python helper; in R, build URLs from the mirror root.
# Mirror failover: see `education-data-query/references/fetch-patterns.md` (R pattern);
# canonical paths: datasets-reference.md.
mirror <- yaml::read_yaml("mirrors.yaml")$mirrors[[1]]

# School-to-census geography (2020 Census boundaries)
# NOTE: illustrative only — mirror parquet files are Polars-written and may
# declare string_view columns, so a plain read can fail under R arrow
# ("cannot handle Array of type <utf8_view>"). Real fetch scripts must use the
# view-safe parquet read from `education-data-query/references/fetch-patterns.md`.
df <- read_parquet(paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "nhgis/schools_nhgis_geog_2020", ".", mirror$format))

# College-to-census geography (2020 Census boundaries)
df_colleges <- read_parquet(paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "nhgis/colleges_nhgis_geog_2020", ".", mirror$format))

# Available census years: 1990, 2000, 2010, 2020
# All files contain ALL data years (schools: 1986-2023, colleges: 1980, 1984-2023)
# The census year determines which boundary vintage is used for geographic assignment
```

**Available data**: Institution coordinates linked to census tract, block group, block, region, division, CBSA, and place. See `variable-catalog.md` for Portal integer encodings.

Codebooks are `.xls` files co-located with data in all mirrors. Use `get_codebook_url()` from `fetch-patterns.md` to construct download URLs. Codebook paths are listed in `datasets-reference.md`.

> **Truth Hierarchy:** When interpreting variable values, apply this priority:
> 1. **Actual data file** (what you observe in the parquet/CSV) — this IS the truth
> 2. **Live codebook** (.xls in mirror) — authoritative documentation, may lag
> 3. **This skill documentation** — convenient summary, may drift from codebook
>
> If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate.

**For custom census analysis** (tract-level demographics, time series, boundary files): Use NHGIS directly via methods below.

---

## Direct NHGIS Access (Custom Analysis)

For full census data access beyond school/college-census links, use NHGIS directly. This is separate from the Education Data Portal mirror system.

### Browsing, Account, and API Requirements

The NHGIS Data Finder at `https://data2.nhgis.org/main` can be browsed without signing in: users can inspect datasets, tables, years, topics, and geographic levels publicly. A free IPUMS NHGIS account and login are required to **submit, manage, and download extracts**. Register at `https://uma.pop.umn.edu/nhgis/user/new`.

API extract requests have an additional requirement: an IPUMS API key associated with the registered account. See `https://developer.ipums.org/docs/v2/get-started/` for current prerequisites.

Authentication therefore applies to extract operations, not to ordinary Data Finder browsing. Account access provides:
- Extract submission, history, management, and downloads
- API-key creation for programmatic extract requests

## Method 1: NHGIS Data Finder (Web Interface)

### URL
https://data2.nhgis.org/main

### Workflow

1. **Filter data** using sidebar options:
   - Geographic Levels (State, County, Tract, etc.)
   - Years (1790-2024 as of the February 2026 ACS release; check the live filter)
   - Topics (Population, Housing, etc.)
   - Datasets (Decennial, ACS periods)

2. **Select data type** (tabs):
   - SOURCE TABLES: Original census tables
   - TIME SERIES TABLES: Harmonized across years
   - GIS FILES: Boundary shapefiles

3. **Add to Data Cart**:
   - Click "+" to add tables/files
   - Review cart contents

4. **Configure extract** (Continue button):
   - Select geographic levels for tables
   - Choose output format (CSV; optional SAS/SPSS/Stata control files)
   - Select layout for time series

5. **Submit extract**:
   - Extracts queued on server
   - Email notification when ready
   - Download from Extracts History page

### Output Formats

| Format | Extension | Best For |
|--------|-----------|----------|
| CSV | .csv | General use, Excel, Python, R |
| Shapefile | .shp | GIS software |

> NHGIS discontinued fixed-width table extracts on 2026-02-11; table data are delivered as CSV only, with optional control files for SAS, SPSS, or Stata (NHGIS Revision History).

### Extract Contents

Each extract includes:
- Data file(s) (.csv)
- Codebook (.txt) - variable descriptions
- For GIS: multiple shapefile components

## Method 2: IPUMS API

### Overview

RESTful API for programmatic data requests.

**Documentation**: https://developer.ipums.org/docs/v2/apiprogram/

### Authentication

```bash
# API key in header
curl -X GET "https://api.ipums.org/metadata/datasets?collection=nhgis&version=2" \
  -H "Authorization: YOUR_API_KEY"
```

### Endpoints

| Endpoint | Purpose |
|----------|---------|
| `/metadata/datasets?collection=nhgis&version=2` | List available datasets |
| `/metadata/data_tables?collection=nhgis&version=2` | List tables |
| `/metadata/time_series_tables?collection=nhgis&version=2` | List time series |
| `/extracts/` | Create and manage extracts |

### Creating Extract via API

```python
import requests

api_key = "YOUR_API_KEY"
headers = {"Authorization": api_key}

# Define extract
extract_def = {
    "datasets": {
        "2016_2020_ACS5a": {
            "dataTables": ["B01001"],
            "geogLevels": ["tract"]
        }
    },
    "dataFormat": "csv_no_header",
    "description": "Population by age, tract level"
}

# Submit extract (IPUMS API Version 2)
response = requests.post(
    "https://api.ipums.org/extracts/?collection=nhgis&version=2",
    headers=headers,
    json=extract_def
)

extract_id = response.json()["number"]
```

```r
library(httr2)

api_key <- "YOUR_API_KEY"

# Define extract
extract_def <- list(
  datasets = list(
    `2016_2020_ACS5a` = list(
      dataTables = list("B01001"),
      geogLevels = list("tract")
    )
  ),
  dataFormat = "csv_no_header",
  description = "Population by age, tract level"
)

# Submit extract
resp <- request("https://api.ipums.org/extracts/?collection=nhgis&version=2") |>
  req_headers(Authorization = api_key) |>
  req_body_json(extract_def) |>
  req_perform()

extract_id <- resp_body_json(resp)$number
```

### Checking Extract Status

```python
status_url = f"https://api.ipums.org/extracts/{extract_id}?collection=nhgis&version=2"
response = requests.get(status_url, headers=headers)
status = response.json()["status"]
# "queued", "started", "produced", "canceled", "failed", "completed"
```

```r
status_url <- paste0(
  "https://api.ipums.org/extracts/", extract_id,
  "?collection=nhgis&version=2"
)
response <- request(status_url) |> req_headers(Authorization = api_key) |> req_perform()
status <- resp_body_json(response)$status
# "queued", "started", "produced", "canceled", "failed", "completed"
```

### Downloading Extract

```python
if status == "completed":
    download_links = response.json()["downloadLinks"]
    for name, link in download_links.items():
        data = requests.get(link["url"], headers=headers)
        # Save to file
```

```r
if (status == "completed") {
  download_links <- resp_body_json(response)$downloadLinks
  for (link in download_links) {
    download.file(link$url, destfile = basename(link$url),
                  headers = c(Authorization = api_key))
  }
}
```

## Method 3: ipumspy (Python)

### Availability

`ipumspy` is **not** pre-installed in DAAF, and runtime installs are blocked
(`pip install` is refused at the command line and inside executed scripts — see
CLAUDE.md § Runtime Package Installation). This access method is therefore
unavailable until `ipumspy` is added to the Dockerfile (user additions block near
the end) and the container is rebuilt. The direct-API methods shown above need no
extra package and work today.

### Configuration

```python
from ipumspy import IpumsApiClient, readers

# Initialize client
client = IpumsApiClient(api_key="YOUR_API_KEY")
```

### Creating and Submitting Extract

```python
from ipumspy.nhgis import NhgisExtract

# Define extract
extract = NhgisExtract(
    datasets={
        "2016_2020_ACS5a": {
            "data_tables": ["B01001", "B19013"],
            "geog_levels": ["tract"]
        }
    }
)

# Submit
client.submit_extract(extract)
print(f"Extract number: {extract.extract_id}")
```

### Waiting and Downloading

```python
# Wait for completion
client.wait_for_extract(extract)

# Download
client.download_extract(extract, download_dir="./data")
```

### Reading NHGIS Data

```python
from ipumspy import readers

# Read tabular data with codebook
ddi = readers.read_nhgis_codebook("nhgis_extract_codebook.txt")
data = readers.read_nhgis("nhgis_data.csv", ddi)
```

### Example: Complete Workflow

```python
import polars as pl
from ipumspy import IpumsApiClient
from ipumspy.nhgis import NhgisExtract

# Setup
client = IpumsApiClient("YOUR_API_KEY")

# Define extract: Poverty data for tracts
extract = NhgisExtract(
    datasets={
        "2018_2022_ACS5a": {
            "data_tables": ["B17001"],  # Poverty status
            "geog_levels": ["tract"]
        }
    },
    geographic_extents=["California"],  # Single state
    data_format="csv_header"
)

# Submit and wait
client.submit_extract(extract)
client.wait_for_extract(extract)

# Download
files = client.download_extract(extract, download_dir="./nhgis_data")

# Read data with Polars
df = pl.read_csv(files["data"][0])
```

## Method 4: ipumsr (R)

### Availability

`ipumsr` is **not** pre-installed in DAAF, and runtime installs are blocked
(`install.packages()` is refused at the command line and inside executed scripts —
see CLAUDE.md § Runtime Package Installation). This access method is therefore
unavailable until `ipumsr` is added to the Dockerfile (user additions block near
the end) and the container is rebuilt. The direct-API methods shown above need no
extra package and work today.

### Configuration

```r
library(ipumsr)

# Set API key (or use environment variable IPUMS_API_KEY)
set_ipums_api_key("YOUR_API_KEY")
```

### Creating Extract

```r
# Define NHGIS extract
extract <- define_extract_nhgis(
  description = "Tract poverty data",
  datasets = ds_spec(
    "2018_2022_ACS5a",
    data_tables = "B17001",
    geog_levels = "tract"
  )
)
```

### Submitting and Downloading

```r
# Submit
submitted <- submit_extract(extract)

# Wait for completion
completed <- wait_for_extract(submitted)

# Download
downloaded <- download_extract(completed, download_dir = "./data")
```

### Reading Data

```r
# Read with variable labels
nhgis_data <- read_nhgis(downloaded)

# Access codebook info
nhgis_cb <- read_nhgis_codebook(downloaded)
```

### Time Series Example

```r
# Time series extract
ts_extract <- define_extract_nhgis(
  description = "Population time series",
  time_series_tables = tst_spec(
    "CL8",  # Total population table
    geog_levels = "county",
    years = c("1990", "2000", "2010", "2020")
  )
)

submitted <- submit_extract(ts_extract)
wait_for_extract(submitted)
downloaded <- download_extract(submitted)
```

## Accessing Geographic Crosswalks

Crosswalk files require a signed-in IPUMS account: `secure-assets.ipums.org` crosswalk links redirect to the IPUMS login page (observed 2026-10-19). They are also available through the IPUMS API (Version 2 supplemental-data endpoints, added October 2024): replace `https://secure-assets.ipums.org/` with `https://api.ipums.org/supplemental-data/` and send the API key in the `Authorization` header.

### Web Download (login required)
https://www.nhgis.org/geographic-crosswalks#download

### Direct URLs (Examples; login or API key required)

```
# 2000 blocks to 2010 tracts
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_blk2000_tr2010.zip

# 2010 tracts to 2020 tracts
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_tr2010_tr2020.zip

# State-specific (faster download)
https://secure-assets.ipums.org/nhgis/crosswalks/nhgis_blk2000_blk2010_state/nhgis_blk2000_blk2010_06.zip  # California
```

## Accessing GIS Files

### Via Data Finder

1. Click "GIS FILES" tab
2. Filter by geographic level and year
3. Add to cart and submit extract

### Via API/ipumspy

```python
# Include shapefiles in extract
extract = NhgisExtract(
    datasets={"2016_2020_ACS5a": {...}},
    shapefiles=["2020_tract"]
)
```

```r
# Include shapefiles in extract (ipumsr)
extract <- define_extract_nhgis(
  description = "Data with shapefiles",
  datasets = ds_spec("2016_2020_ACS5a",
    data_tables = "B01001",
    geog_levels = "tract"
  ),
  shapefiles = "2020_tract"
)
```

### Direct TIGER/Line (Alternative)

For current boundaries only:
https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html

**Note**: NHGIS shapefiles are processed (clipped to coastlines, consistent projection).

## Data Format Details

### CSV Output

```csv
GISJOIN,YEAR,STATE,STATEA,COUNTY,COUNTYA,TRACTA,AJWME001,AJWME002
G0600010000100,2020,California,06,Alameda,001,000100,3245,1587
```

### Key Fields

| Field | Description |
|-------|-------------|
| GISJOIN | NHGIS geographic identifier |
| YEAR | Data year |
| STATE | State name |
| STATEA | State FIPS (with NHGIS suffix) |
| COUNTY | County name |
| COUNTYA | County FIPS (with NHGIS suffix) |
| [Variable codes] | Data values |

### Codebook Format

```
Context Fields 
    GISJOIN:     GIS Join Match Code
    YEAR:        Data File Year
    ...

Table 1:     B01001.    Sex by Age
Universe:    Total population
    AJWME001:    Total
    AJWME002:    Male
    AJWME003:    Male: Under 5 years
    ...
```

## Performance Tips

### Large Extracts

- Select specific states (geographic extent) when possible
- Request only needed geographic levels
- Request only the tables and years you need (fixed-width output was discontinued 2026-02-11)

### Efficient API Usage

```python
# Check if extract exists before resubmitting
existing = client.get_extract_by_id(extract_id, collection="nhgis")
if existing["status"] == "completed":
    # Download existing
    pass
else:
    # Submit new
    pass
```

```r
# Check if extract exists before resubmitting (ipumsr)
existing <- get_extract_info(submitted)
if (existing$status == "completed") {
  # Download existing
} else {
  # Submit new
}
```

### Batch Processing

```python
# Submit multiple extracts
extracts = []
for state in ["California", "Texas", "New York"]:
    ext = NhgisExtract(
        datasets={"2016_2020_ACS5a": {...}},
        geographic_extents=[state]
    )
    client.submit_extract(ext)
    extracts.append(ext)

# Wait for all
for ext in extracts:
    client.wait_for_extract(ext)
```

```r
# Submit multiple extracts (ipumsr)
states <- c("California", "Texas", "New York")
extracts <- lapply(states, function(st) {
  ext <- define_extract_nhgis(
    description = paste("Data for", st),
    datasets = ds_spec(
      "2016_2020_ACS5a",
      data_tables = "B01001",
      geog_levels = "tract"
    ),
    geographic_extents = st
  )
  submit_extract(ext)
})

# Wait for all
completed <- lapply(extracts, wait_for_extract)
```

## Troubleshooting

| Issue | Cause | Solution |
|-------|-------|----------|
| Extract stuck "queued" | Server load | Wait; check status |
| "Unauthorized" error | Invalid API key | Regenerate key in account settings |
| Empty data file | Invalid table/level combo | Check Data Finder for valid combinations |
| Missing variables | Table not available for level | Choose different geographic level |
| Shapefile mismatch | Year mismatch | Ensure shapefile year matches data year |

## Terms, Citation, and Redistribution

Direct NHGIS extracts are governed by IPUMS terms, citation requirements, and redistribution restrictions that are separate from the Education Data Portal's ODC-By terms. Review the current requirements before sharing an extract or derivative:

- Citation and use: `https://www.nhgis.org/citation-and-use-nhgis-data`
- IPUMS terms: `https://www.ipums.org/about/terms`

As verified on 2026-07-21, the citation page identifies **IPUMS NHGIS Version 21.0 (2026)**. Use the exact citation generated for the extract or shown on the current citation page rather than copying a frozen version string from this reference.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall