brycewang-stanford/Auto-Empirical-Research-Skills / skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/education-data-source-ccd
education-data-source-ccd
VerifiedCCD — federal universe of all U.S. public K-12 schools (~100K) and districts (~18K). Enrollment, staffing, finance, directory data (1986-present). Use for public school analysis by grade/race/sex. Public only; excludes private and postsecondary.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.3 KB |
| references/data-collection.md | markdown | 8.6 KB |
| references/data-quality.md | markdown | 10.9 KB |
| references/historical-changes.md | markdown | 12.5 KB |
| references/survey-components.md | markdown | 14.8 KB |
| references/variable-definitions.md | markdown | 17.3 KB |
references/historical-changes.md
View on GitHub# CCD Historical Changes
This reference documents significant changes to CCD data collection, definitions, and coding over time. Essential for longitudinal analysis and understanding data comparability.
> **Portal vs NCES Codes:** This document references historical NCES string codes (AM, BL, PK, etc.) when explaining transitions. The Education Data Portal uses **integer codes** — see `variable-definitions.md` for current Portal mappings.
## Major Milestones Timeline
| Year | Change | Impact |
|------|--------|--------|
| 1986-87 | CCD nonfiscal data begins | Historical baseline |
| 1989-90 | Finance data collection begins | District finance available |
| 1991-92 | Dropout/completer collection begins | Completion data available |
| 1998-99 | English Language Learner variable added | ELL tracking starts |
| 2002-03 | School status codes 6, 7 added | Inactive/future schools tracked |
| 2005-06 | School status code 8 added | Reopened schools tracked |
| 2006-07 | **Locale codes revised** | Major discontinuity |
| 2006 | EDFacts system implemented | Data collection centralized |
| 2007-08 | LEA type codes revised | Charter agencies distinguished |
| 2007-08 | School type code 5 added | Reportable programs tracked |
| 2007 | ACGR introduced | Graduation rate methodology change |
| 2010-11 | **Race/ethnicity categories revised** | Major discontinuity |
| 2014-15 | Virtual school indicator added | Virtual schools tracked |
| 2014-15 | CEP nationwide | FRPL interpretation affected |
| 2016-17 | Long file format | Data structure changed |
| 2017-18 | LEA membership guidance changed | LEA totals methodology |
| 2017-18 | School level assignment logic changed | LEVEL variable affected |
---
## Locale Codes (2006)
### The Change
In 2006, NCES completely revised the locale code system from a "metro-centric" to an "urban-centric" framework.
### Before 2006: Metro-Centric Codes
| Code | Category |
|------|----------|
| 1 | Large city |
| 2 | Midsize city |
| 3 | Urban fringe of large city |
| 4 | Urban fringe of midsize city |
| 5 | Large town |
| 6 | Small town |
| 7 | Rural, outside CBSA |
| 8 | Rural, inside CBSA |
### After 2006: Urban-Centric Codes
| Code | Category | Definition |
|------|----------|------------|
| 11 | City, Large | Principal city, pop ≥250,000 |
| 12 | City, Midsize | Principal city, pop 100,000-249,999 |
| 13 | City, Small | Principal city, pop <100,000 |
| 21 | Suburb, Large | Outside principal city, UA ≥250,000 |
| 22 | Suburb, Midsize | Outside principal city, UA 100,000-249,999 |
| 23 | Suburb, Small | Outside principal city, UA <100,000 |
| 31 | Town, Fringe | Urban cluster, ≤10 mi from UA |
| 32 | Town, Distant | Urban cluster, 10-35 mi from UA |
| 33 | Town, Remote | Urban cluster, >35 mi from UA |
| 41 | Rural, Fringe | ≤5 mi from UA |
| 42 | Rural, Distant | 5-25 mi from UA |
| 43 | Rural, Remote | >25 mi from UA |
### Why There's No Crosswalk
The systems use fundamentally different geographic concepts:
- Old: Based on Metropolitan Statistical Areas (MSAs)
- New: Based on urbanized areas and distance calculations
A school classified as "rural" in the old system might be "town" or "suburb" in the new system, and vice versa.
### Recommendations for Longitudinal Analysis
1. **Preferred**: Analyze pre-2006 and post-2006 as separate periods
2. **Alternative**: Use broad categories (urban/suburban/town/rural) with caution
3. **Advanced**: Obtain geocodes and calculate custom locale measures
### Transition Files
NCES provides files for 2003-04 through 2005-06 that include both old and new codes, useful for understanding the transition.
---
## Race/Ethnicity Categories (2010)
### The Change
In 2010, CCD adopted new OMB standards for race/ethnicity reporting, adding a "Two or More Races" category and separating Asian and Pacific Islander.
### Before 2010: 5 Categories
| Code | Category |
|------|----------|
| AI | American Indian/Alaska Native |
| AS | Asian/Pacific Islander (combined) |
| BL | Black, non-Hispanic |
| HI | Hispanic |
| WH | White, non-Hispanic |
### After 2010: 7 Categories
| Code | Category |
|------|----------|
| HI | Hispanic/Latino (any race) |
| AM | American Indian/Alaska Native (non-Hispanic) |
| AS | Asian (non-Hispanic) |
| BL | Black or African American (non-Hispanic) |
| HP | Native Hawaiian/Pacific Islander (non-Hispanic) |
| TR | Two or More Races (non-Hispanic) |
| WH | White (non-Hispanic) |
### Key Differences
1. **Two-question format**: Hispanic ethnicity asked separately from race
2. **Two or More Races**: New category for multiracial students
3. **Asian/Pacific Islander split**: Separate categories for AS and HP
4. **Collection order**: Hispanic status collected first
### Transition Period (2007-2010)
States transitioned at different times:
- Some early adopters in 2007-08
- Most states by 2009-10
- Full implementation by 2010-11
**Data Issues During Transition**:
- Inconsistent categories across states
- Some students in "unknown" race
- Bridge coding attempts (mapping old to new)
### Impact on Time Series
| Analysis Need | Impact | Approach |
|---------------|--------|----------|
| Total by race | Can combine HP with AS for historical comparison | Loses HP detail |
| Multiracial students | TR category didn't exist pre-2010 | Cannot track historically |
| Hispanic students | Relatively consistent | Generally comparable |
| Detailed race analysis | Not comparable across periods | Separate analyses |
### Recommended Approach
```python
# Portal uses integer race codes (not NCES string codes)
# Current Portal codes: 1=White, 2=Black, 3=Hispanic, 4=Asian,
# 5=AIAN, 6=NHPI, 7=Two+, 99=Total
# For time series spanning 2010
# Option 1: Collapse new categories to match old 5-category system
def collapse_to_5_categories(race_code: int) -> int:
"""Collapse 7-category (post-2010) to 5-category (pre-2010) equivalent."""
mapping = {
5: 5, # American Indian/Alaska Native stays same
4: 4, # Asian stays Asian
6: 4, # Pacific Islander → Asian (combined pre-2010)
2: 2, # Black stays Black
3: 3, # Hispanic stays Hispanic
1: 1, # White stays White
7: -1, # Two or more → cannot map (treat as missing)
}
return mapping.get(race_code, -1)
# Option 2: Analyze pre-2010 and post-2010 separately
# Note the break in your time series documentation
```
---
## LEA Type Codes (2007)
### The Change
In 2007-08, LEA type codes were revised to specifically identify charter school agencies.
### Before 2007-08
| Code | Type |
|------|------|
| 1-6 | Various types (same as current) |
| 7 | Other education agencies |
### After 2007-08
| Code | Type |
|------|------|
| 1-6 | Same as before |
| 7 | **Charter agency** (all schools are charters) |
| 8 | Other education agencies |
### Impact
- Pre-2007-08: Charter agencies coded as "7" (Other) or regular districts
- Post-2007-08: Charter agencies specifically identifiable
- Historical charter LEA identification requires additional logic
### Identifying Historical Charter LEAs
For years before 2007-08:
1. Check if all schools have charter=1
2. Cross-reference with state charter authorizer records
3. Document uncertainty in analysis
---
## School Type Codes (2007)
### The Change
In 2007-08, school type code 5 was added for "Reportable Programs."
### Before 2007-08
| Code | Type |
|------|------|
| 1 | Regular |
| 2 | Special Education |
| 3 | Vocational |
| 4 | Alternative |
### After 2007-08
| Code | Type |
|------|------|
| 1-4 | Same as before |
| 5 | Reportable Program (within another school) |
### Impact
Programs that were previously reported as separate schools may now be coded as programs. This affects school counts.
---
## Status Codes
### Additions Over Time
| Year | Code | Meaning |
|------|------|---------|
| Original | 1-5 | Continuing, Closed, New, Added, Changed Agency |
| 2002-03 | 6 | Inactive (temporarily closed, may reopen ≤3 years) |
| 2002-03 | 7 | Future (scheduled to open within 2 years) |
| 2005-06 | 8 | Reopened (was closed, now operational) |
### Impact on School Counts
When comparing school counts over time:
- Pre-2002-03: Temporarily closed schools may appear as "Closed"
- Post-2002-03: "Inactive" status distinguishes temporary closures
- Pre-2005-06: Reopened schools counted as "New"
- Post-2005-06: "Reopened" status tracks these separately
---
## Graduation Rate Methodology (2007)
### AFGR (Averaged Freshman Graduation Rate)
**Used**: Before ACGR implementation
**Calculation**:
```
AFGR = Diploma Recipients / Averaged 9th Grade Enrollment
Where averaged enrollment = (8th grade y-4 + 9th grade y-3 + 9th grade y-2) / 3
```
**Limitations**:
- Uses enrollment approximation, not cohort tracking
- Doesn't account for transfers
- State implementation varied
### ACGR (Adjusted Cohort Graduation Rate)
**Required**: Under ESEA/ESSA
**Calculation**:
```
ACGR = (Cohort who graduated) / (First-time 9th graders, adjusted for transfers)
```
**Features**:
- Tracks actual student cohorts
- Adjusts for documented transfers in/out
- Standardized methodology
### Transition
- ACGR introduced 2007
- Phased in through 2011
- Both measures available during transition
- ACGR is now the standard measure
---
## File Format Change (2016-17)
### The Change
Starting 2016-17, CCD files are published in "long" format instead of "wide" format.
### Wide Format (Pre-2016-17)
One row per school, columns for each variable:
```
NCESSCH, AM_MALE_K, AM_MALE_1, ..., WH_FEMALE_12
```
### Long Format (2016-17+)
Multiple rows per school, disaggregation as columns:
```
NCESSCH, GRADE, SEX, RACE, STUDENT_COUNT
```
### Impact
- Easier to filter/aggregate
- Different data processing required
- NCES provides conversion programs
---
## Membership Reporting Guidance (2017-18)
### The Change
Guidance for reporting LEA-level membership changed between 2016-17 and 2017-18.
### Before 2017-18
States could report LEA totals that:
- Were independent of school sums
- Used different counting methods
- Might not equal sum of schools
### After 2017-18
LEA membership should equal sum of school membership within the LEA.
### Impact
Comparing LEA-level membership across this transition requires caution. Some states show apparent changes that reflect reporting methodology, not actual enrollment changes.
---
## School Level Assignment (2017-18)
### The Change
The logic for deriving the LEVEL variable (elementary, middle, high, other) changed in 2017-18.
### Impact
Some schools may be classified differently before and after this change. Check school-level time series for unexpected changes.
---
## FRPL Reporting Changes
### Timeline
| Year | Change |
|------|--------|
| Original | FRPL reported from applications |
| 2011-12 | CEP pilot begins (some states) |
| 2012-13 | Direct certification reported separately (optional) |
| 2014-15 | CEP nationwide |
| 2016-17+ | States may report FRPL, Direct Cert, or both |
### Impact on Poverty Analysis
- Pre-CEP: FRPL reasonably consistent poverty proxy
- Post-CEP: Schools with 100% FRPL may be CEP schools, not 100% poverty
- Current: Must interpret FRPL differently for CEP vs. non-CEP schools
---
## Handling Historical Changes in Analysis
### General Approach
1. **Document the relevant period** for your analysis
2. **Check for definition changes** during your period
3. **Account for changes** in methodology or interpretation
4. **Report limitations** clearly
### Code Pattern for Time-Sensitive Analysis
```python
def prepare_longitudinal_data(df, variable, start_year, end_year):
"""Prepare data accounting for known historical changes."""
# Check for locale code issues
if variable == "locale" and start_year < 2006 and end_year >= 2006:
print("WARNING: Locale code methodology changed in 2006")
print("Consider separate pre/post-2006 analysis")
# Check for race/ethnicity issues
if variable in ["race", "enrollment_by_race"] and start_year < 2010 and end_year >= 2010:
print("WARNING: Race/ethnicity categories changed in 2010")
print("Two or More Races (TR) and HP categories not available pre-2010")
# Check for FRPL issues
if variable in ["frpl", "free_lunch", "reduced_lunch"] and end_year >= 2014:
print("NOTE: CEP implementation affects FRPL interpretation post-2014")
return df
```
---
## Resources for Historical Research
### NCES Documentation
- Reference Library: Crosswalks and documentation
- Historical file documentation
- State notes by year
### Useful Publications
- "100 Years of American Education: A Statistical Portrait" (NCES)
- CCD overview documents
- Digest of Education Statistics (annual)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |