DAAF-Contribution-Community/daaf / .claude/skills/education-data-explorer
education-data-explorer
UnverifiedMirror-first discovery for Urban Institute Education Data: maps a research question to the mirror dataset FILES that answer it — canonical path, entity grain, year coverage, key variables, and join keys — for CCD, IPEDS, CRDC, College Scorecard, SAIPE, EDFacts, FSA, MEPS, NHGIS, PSEO, NACUBO, NCCS, EADA, and Campus Safety. Discovery output is the mirror path (plus grain, years, variables, caveats), the currency planning and acquisition consume. A pure readable-reference skill for read-only discovery agents — no code, no live HTTP probes; all facts keyed to pinned mirror vintage 0.26.1. Load BEFORE education-data-query — routing (which file, what grain) here, retrieval (fetch code, mirror config) there.
Education Data Explorer
Discovers which Urban Institute Education Data mirror files answer a research question and returns them as acquisition-ready routing facts: the canonical mirror path, what one row represents (grain), year coverage, curated key variables, and join keys. This is the discovery half of the education-data workflow — it owns question→file routing judgment, grain semantics, variable/code meaning, and cross-source joins. It does not own fetch mechanics: retrieval code, mirror configuration, codebook-URL construction, and the curated path catalog live in education-data-query. Load this skill first (routing), then education-data-query (retrieval). Every inventory fact here — years, variables, coded values — is keyed to mirror vintage 0.26.1 (pinned); the discovery path is pure markdown reading, with no instruction to fetch, probe, or execute anything.
Why mirror-first (and why no API)
DAAF's education pipeline never touches Urban's live REST API. All acquisition is mirror-based: a revision-pinned HuggingFace parquet mirror (Portal vintage 0.26.1, revision 0ad00ce0e232c96b0642459e4e7326607a8d26aa), with an unpinned urban_csv bulk-CSV fallback that uses the identical canonical paths. Acquisition consumes canonical mirror paths of the form {source}/{filename} (extensionless — the mirror layer appends .parquet or .csv). So discovery's job is to emit those paths, not URL routes. Portal endpoint routes appear here only as small citation/maintenance annotations, never as the thing you hand downstream.
The live REST API moves quarterly (endpoints rename, variables drift); a prior version of this skill hand-documented that API and went badly stale. The mirror is a frozen, reproducible snapshot — so the facts below are stable for vintage 0.26.1 and can be read as reference rather than re-probed.
Standing caveat (vintage + fallback). All year coverage, variables, and coded values below are exact for the pinned HuggingFace mirror at vintage 0.26.1. The
urban_csvfallback is unpinned/current-Portal, so if a fetch falls through to it, vintage guarantees no longer hold exactly. Cite Portal version 0.26.1 for data fetched from the pinned mirror.
Reference File Structure
| File | Purpose | When to Read |
|---|---|---|
references/schools-datasets.md | K-12 school-level mirror families (CCD, CRDC, EDFacts, MEPS, NHGIS) | Researching individual schools |
references/districts-datasets.md | District/LEA-level mirror families (CCD, SAIPE, EDFacts) | Researching school districts |
references/colleges-datasets.md | College-level mirror families (IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety) | Researching higher education |
references/variable-dictionary-{source}.md | Complete per-source variable inventory (2,235 per-source variables, deduplicated from 2,994 endpoint-variable rows; verbatim labels/types/filter flags/coded values) | Confirming a real variable name or its coded values |
references/variable-codes.md | Curated, analytically load-bearing coded-value schemes with cross-source populated-subset notes | Interpreting or filtering categorical codes |
references/maintenance-live-api.md | Live-API catalog surfaces + regeneration mechanics — maintenance sessions only | Rebuilding the mirror/dictionaries at a new vintage |
The 14
variable-dictionary-{source}.mdfiles are mechanically generated from the Portal varlist at vintage 0.26.1. The routing files curate 5-12 key variables per family and point to the full dictionary entry for the rest. Never invent a variable name — if it is not in the dictionary, it is not in the mirror.
Decision Trees
What data level (and file) do I need?
What entity is one row of my analysis?
├─ An individual K-12 school (ncessch) → schools level
│ └─ references/schools-datasets.md → pick a CCD/CRDC/EDFacts/MEPS/NHGIS family
├─ A school district / LEA (leaid) → school-districts level
│ └─ references/districts-datasets.md → pick a CCD/SAIPE/EDFacts family
└─ A college / university (unitid) → college-university level
└─ references/colleges-datasets.md → pick an IPEDS/Scorecard/FSA/PSEO/EADA/NACUBO/NCCS/csafety family
What topic → which mirror family?
Research topic?
├─ K-12 enrollment / demographics → ccd/schools_ccd_enrollment_{year} (or ccd/schools_ccd_lea_enrollment_{year} at district level)
├─ School/district directory & staffing → ccd/schools_ccd_directory | ccd/school-districts_lea_directory
├─ District finance → ccd/districts_ccd_finance
├─ District poverty → saipe/districts_saipe
├─ Civil rights / discipline / course access → crdc/schools_crdc_* (see schools-datasets.md — one file per topic)
├─ K-12 assessments / grad rates → edfacts/{schools,districts}_edfacts_assessments_{year} | _grad_rates_{year}
├─ School poverty measure → meps/schools_meps
├─ College directory / characteristics → ipeds/colleges_ipeds_directory | _institutional-characteristics
├─ College enrollment → ipeds/colleges_ipeds_fall-enrollment-race_{year} | _enrollment-fte | _headcount
├─ College completions (by CIP) → ipeds/colleges_ipeds_completions-2digcip_{year} | -6digcip_{year}
├─ College graduation rates → ipeds/colleges_ipeds_grad-rates | -200pct | -pell
├─ College finance / endowment → ipeds/colleges_ipeds_finance | nacubo/colleges_nacubo_endow | nccs/colleges_nccs_all
├─ Student aid / loans → ipeds/colleges_ipeds_sfa_* | fsa/colleges_fsa_grants | _loans | _campus_based_volume
├─ Post-college earnings → scorecard/colleges_scorecard_earnings | pseo/colleges_pseo_{year}
├─ Athletics equity → eada/colleges_eada_inst_characteristics
├─ Campus crime (hate crimes) → csafety/colleges_csafety_hate_crimes
└─ Census geography crosswalk → nhgis/{schools,colleges}_nhgis_geog_{census-year}
How do I confirm a variable name or code?
Need a variable?
├─ Curated key variables → the family entry in the *-datasets.md routing file
├─ Every variable in a family → references/variable-dictionary-{source}.md (§ that family)
└─ What a categorical code means → references/variable-codes.md
Quick Reference: Data Levels
| Level | Key Sources | Primary ID | ID Format |
|---|---|---|---|
| schools | CCD, CRDC, EDFacts, MEPS, NHGIS | ncessch | 12-char zero-padded string |
| school-districts | CCD, SAIPE, EDFacts | leaid | 7-char zero-padded string |
| college-university | IPEDS, Scorecard, FSA, PSEO, EADA, NACUBO, NCCS, NHGIS, Campus Safety | unitid | 6-digit integer |
Zero-padded IDs:
ncessch(12) andleaid(7) are zero-padded numeric strings — leading zeros are load-bearing (FIPS 01-09 states).education-data-queryhandles the force-string/pad-and-assert read recipe; discovery just needs to name the join key. Raw mirror dtypes differ by file:ncessch/leaidare String in some files (e.g.ccd/schools_ccd_directory,crdc/schools_crdc_apib_enroll) and Int64 without leading zeros in others (e.g.meps/schools_meps,ccd/schools_ccd_enrollment_{year}before 2024,saipe/districts_saipe); a Polars join across the two raises a SchemaError (pinned mirror, probed 2026-10-28). Emit the dtype normalization as a caveat with the join key.
Quick Reference: Join Keys
| Join across | Key | Example use |
|---|---|---|
| School sources (CCD ↔ CRDC ↔ EDFacts ↔ MEPS ↔ NHGIS) | ncessch (+ year) | Enrollment + discipline + poverty |
| District sources (CCD ↔ SAIPE ↔ EDFacts) | leaid (+ year) | Directory + poverty + outcomes |
| College sources (IPEDS ↔ Scorecard ↔ FSA ↔ PSEO ↔ EADA ↔ NACUBO ↔ NCCS ↔ csafety) | unitid (+ year; PSEO repeats each cohort under every year in its span, so de-duplicate first) | Characteristics + earnings + aid |
| School → district rollup | leaid present on school files | Aggregate schools to their LEA |
| Any level → census geography | ncessch/unitid via NHGIS crosswalk | Link to tract/CBSA/region |
Match on year when joining — coverage windows differ across families (see below), so align years before merging.
Year Coverage Overview (vintage 0.26.1)
Verified ranges from the endpoint-file bridge and route reconciliation. Non-contiguous families are flagged; see the routing files for exact year lists.
| Family (representative) | Coverage | Note |
|---|---|---|
| CCD directory / enrollment | 1986-2024 | Annual |
| CCD district finance | 1989, 1991, 1994-2020 | Non-contiguous early years; 1989 is in the mirror file but not in the Portal catalog |
| CRDC (most topics) | 2011, 2013, 2015, 2017, 2020, 2021 | Often called biennial, but no odd/even rule (2020 and 2021 are consecutive); chronic-absenteeism reaches 2022 |
| EDFacts assessments | 2009-2018, 2020 | 2019 absent (COVID testing waivers) |
| EDFacts grad rates | 2010-2019 | — |
| IPEDS directory / inst-characteristics | 1980, 1984-2024 | — |
| IPEDS finance | 1979, 1983-2017 | Mirror ends 2017; live Portal API lists 1979, 1983-2023 (endpoint metadata fetched 2026-10-19) — mirror-vintage lag |
| IPEDS grad-rates (150%) | 1996-2023 | Wholesale-revalued in v0.26.1 — re-run any pre-2026q3 analysis |
| IPEDS completions (2-/6-dig CIP) | 1991-2023 / 1983-2023 | Yearly shards |
| Scorecard earnings | 2003-2014, 2018 | Sparse; other Scorecard families 1996-2020 |
| SAIPE | 1995, 1997, 1999-2024 | Poverty pct on a 0-1 scale despite _pct |
| FSA | 1999-2021 | 90/10 from 2014; campus-based from 2001 |
| MEPS | 2009-2022 | 2023 empty on 2026-08-06 probe |
| NHGIS (schools / colleges) | 1986-2023 / 1980, 1984-2023 | Crosswalk file-years; census vintages 1990-2020 |
| PSEO | 2001-2021 | Yearly shards |
| NACUBO / NCCS / EADA / Campus Safety | 2012-2022 / 1993-2016 / 2002-2021 / 2005-2021 | — |
Grain Logic: What a Row Represents
A mirror file's grain is the set of columns that uniquely identify a row. Reading it correctly is the difference between a clean join and a silent fan-out.
- Base grain is the entity ID +
year(e.g., one row perncessch×yearin CCD directory). - Disaggregation dimensions multiply rows. In the live API these appear as URL path segments (
.../enrollment/{year}/{grade}/race/sex/); in the mirror file the same dimensions are columns (grade,race,sex) and each combination is its own row. So CCD enrollment is one row perncessch×year×grade×race×sex. To get a school total, filter to the99/total code on each dimension rather than summing (summing double-counts the totals rows). - CRDC files carry
disabilityandlepdimensions in addition torace/sex; a discipline file is one row per school × year × the disaggregation the file name advertises. - Yearly vs Single files. "Yearly" families have one file per year with
{year}in the path (fetch each year, concatenate). "Single" families pack all years into one file (filter onyearlocally). The routing files mark each family Single or Yearly. - Coded totals. Categorical dimensions use
99= Total and negative sentinels for missing (-1/-2/-3/-99) — seevariable-codes.md. A "total" row and its component rows coexist in the same file. That sentinel legend is the Portal's declared convention, not a guarantee about any given mirror file: a captured probe ofdistricts_saipe(2026-09-15, 382,099 rows) found zero negative sentinels and plain nulls instead. Test for both null and sentinel missingness, and defer to each source skill's documented convention over this general rule.
What Discovery Should Emit (the acquisition currency)
Discovery output feeds planning, then acquisition. For each dataset a research question needs, emit:
- Canonical mirror path(s) —
{source}/{filename}, extensionless, with{year}placeholder where the family is Yearly. This is the primary currency. - Grain — what one row represents, including disaggregation dimensions.
- Years — the specific years needed, within the family's coverage window.
- Key variables — the real column names (from the routing file / dictionary) the analysis will use.
- Join keys — the ID(s) linking this file to the others in the plan.
- Caveats — load-bearing warnings (suppression, revaluation, scale, coverage gaps).
Portal endpoint routes are optional citation annotations only — never the hand-off artifact. Do not instruct downstream to probe, fetch, or call an API; naming the path and years is sufficient for education-data-query to construct the fetch.
Cross-Reference to Related Skills
| Skill | Purpose | When |
|---|---|---|
education-data-query | Fetch code, mirror config, codebook URLs, path catalog | After identifying paths/years/variables here |
education-data-context | Portal-wide interpretation (missing codes, grade encoding, licensing, joins) | Before analyzing retrieved data |
education-data-source-{ccd,ipeds,crdc,scorecard,saipe,edfacts,fsa,meps,nhgis,nccs,nacubo,eada,campus-safety,pseo} | Deep source context: methodology, limitations, historical changes | When you need collection methodology or source-specific caveats beyond routing |
Load a education-data-source-* skill when a question needs collection methodology, known limitations, or historical coding changes for a specific source. Each source skill carries a skill-last-updated provenance date — flag stale ones to the orchestrator.
Worked Example: Planning a Discovery Output
Question: "Relationship between school poverty and AP course offerings in California high schools?"
- Level / grain: schools — one row per
ncessch×year. - Files:
crdc/schools_crdc_apib_enroll(Single; 2011, 2013, 2015, 2017, 2020, 2021) — AP/IB enrollment. Key vars:enrl_ap,enrl_ap_math,enrl_ap_science,enrl_ib,enrl_gifted_talented. Grain addsrace/sex/disability/lepcolumns — filter to totals for a school-level measure.meps/schools_meps(Single; 2009-2022) — modeled poverty. Key var:meps_poverty_pct.
- Join:
ncessch+yearafter normalizing both to a 12-character zero-padded string —crdc/schools_crdc_apib_enrollstores String,meps/schools_mepsInt64, so cast each side first (pl.col("ncessch").cast(pl.Utf8).str.zfill(12)) and assert every key has length 12; overlap years 2011, 2013, 2015, 2017, 2020, 2021. - Filter:
fips == 6(California), high schools via CCDschool_level == 3(joinccd/schools_ccd_directory). - Emit: the three paths, the year window, the key variables, the
ncesschjoin key with its dtype-normalization caveat, and the CRDC disaggregation-totals caveat. Then loadeducation-data-queryto fetch.
Common Pitfalls
- Emitting a route instead of a path. Downstream needs
crdc/schools_crdc_discipline_k12_{year}, not/schools/crdc/discipline/{year}/.... - Summing disaggregated rows. Total rows (
99) coexist with component rows — filter, don't sum. - Assuming a variable exists. If it is not in
variable-dictionary-{source}.md, it is not in the mirror. - Ignoring the coverage window. Align
yearacross families before joining; several have gaps (EDFacts 2019, IPEDS finance ends 2017). - Trusting fallback vintage. Facts are exact for the pinned mirror only; the
urban_csvfallback is current-Portal.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |