DAAF-Contribution-Community/daaf / .claude/skills/education-data-source-edfacts
education-data-source-edfacts
VerifiedEDFacts — K-12 outcomes: assessment proficiency, ACGR graduation rates, ESSA accountability at school/district level (2009-2020). Within-state trends and subgroup gaps. Complements CCD with outcome data. Cannot compare across states — use NAEP.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 21.1 KB |
| references/accountability-context.md | markdown | 9.6 KB |
| references/assessment-data.md | markdown | 15.5 KB |
| references/data-quality.md | markdown | 17.7 KB |
| references/graduation-rates.md | markdown | 18.4 KB |
| references/subgroup-reporting.md | markdown | 21.1 KB |
| references/variable-definitions.md | markdown | 23.8 KB |
SKILL.md
View on GitHub---
name: education-data-source-edfacts
description: >-
EDFacts — K-12 outcomes: assessment proficiency, ACGR graduation rates, ESSA accountability at school/district level (2009-2020). Within-state trends and subgroup gaps. Complements CCD with outcome data. Cannot compare across states — use NAEP.
metadata:
audience: any-agent
domain: data-source
skill-authored: "2026-02-09"
skill-last-updated: "2026-02-09"
---
# EDFacts Data Source Reference
EDFacts — federal K-12 outcome data from State Education Agencies, covering state assessment proficiency rates, ACGR graduation rates, and ESSA accountability indicators at school and district level (assessments 2009-2020, graduation rates 2010-2019). Use when analyzing within-state achievement trends, subgroup proficiency gaps, or adjusted cohort graduation rates. Complements CCD (school characteristics) with outcome data. State assessment scores CANNOT be compared across states; use NAEP for cross-state comparisons.
EDFacts is the U.S. Department of Education's centralized data collection system for pre-K through grade 12 education data from State Education Agencies (SEAs). It provides state assessment proficiency rates, graduation rates, and accountability indicators — the authoritative federal source for state-level K-12 outcome data.
> **CRITICAL: Value Encoding**
>
> The Urban Institute Education Data Portal converts NCES string codes (e.g., `ALL`, `CWD`, `LEP`) to **integer codes**. Always verify actual data values before filtering — do not rely on documentation labels alone.
>
> | Context | Subgroup "All" | English Learner | Sex "Male" |
> |---------|----------------|-----------------|------------|
> | **Portal integer** | `99` | `1` | `1` |
> | NCES string | `ALL` | `LEP` | `M` |
>
> See `./references/variable-definitions.md` for complete encoding tables.
## What is EDFacts?
- **Collector**: U.S. Department of Education, via State Education Agencies (SEAs)
- **Coverage**: All public schools and districts in 50 states + DC
- **Content**: State assessment proficiency rates, ACGR graduation rates, participation rates (in ED's own files, not the Portal), accountability indicators
- **Frequency**: Annual collection
- **Available years (Portal)**: Assessments 2009–2018 and 2020 (no 2019); Graduation rates 2010–2019. For assessments, Portal `year` is the fall of the school year (2019 = SY 2019-20, 2020 = SY 2020-21)
- **Primary identifiers**: `ncessch` (school ID, Int64), `leaid` (district ID, Int64), `fips` (state FIPS code, Int64)
- **Key limitation**: State assessment scores CANNOT be compared across states (different tests, different cut scores)
- **Available through**: Education Data Portal mirrors
## Reference File Structure
| File | Purpose | When to Read |
|------|---------|--------------|
| `accountability-context.md` | ESSA, NCLB history, accountability systems | Understanding policy context |
| `assessment-data.md` | Proficiency levels, test scores, limitations | Working with assessment data |
| `graduation-rates.md` | ACGR methodology, cohort definitions | Analyzing graduation data |
| `variable-definitions.md` | Key variables, suppression codes, special values | Interpreting specific variables |
| `data-quality.md` | Known issues, state variations, COVID impacts | Data cleaning, limitations |
| `subgroup-reporting.md` | Special populations, disaggregation | Analyzing by student groups |
## Decision Trees
### What type of analysis?
```
What EDFacts data do you need?
├─ Assessment/proficiency data
│ ├─ Within-state trends → Valid analysis
│ ├─ Cross-state comparison → INVALID - use NAEP instead
│ └─ Subgroup gaps → See ./references/subgroup-reporting.md
├─ Graduation rates (ACGR)
│ ├─ Understand methodology → See ./references/graduation-rates.md
│ ├─ Extended rates (5-year, 6-year) → See ./references/graduation-rates.md
│ └─ Subgroup rates → See ./references/subgroup-reporting.md
├─ Understanding variables
│ ├─ Missing/suppressed values → See ./references/variable-definitions.md
│ ├─ Range vs. exact values → See ./references/variable-definitions.md
│ └─ Subgroup codes → See ./references/subgroup-reporting.md
└─ Data quality concerns
├─ COVID-19 impacts (2019-20) → See ./references/data-quality.md
├─ State reporting changes → See ./references/data-quality.md
└─ Suppression rates → See ./references/data-quality.md
```
### Is my comparison valid?
```
What are you comparing?
├─ Same state, different years
│ ├─ Same assessment system? → Valid
│ └─ Different tests? → Break in time series
├─ Schools within same state → Valid
├─ Districts within same state → Valid
├─ Subgroups within same school → Valid (check suppression)
├─ Different states
│ ├─ Proficiency rates → INVALID
│ ├─ Graduation rates (ACGR) → More comparable
│ └─ Use NAEP instead → Valid
└─ National ranking by proficiency → INVALID
```
## Quick Reference: EDFacts Data Elements
### Assessment Data
| Data Element | Description | Available Years |
|--------------|-------------|-----------------|
| Proficiency rates | % meeting state standards in reading/math | Portal years 2009–2018, 2020 (no 2019) |
| Participation rates | % of students assessed | Not in the Portal assessment files (no participation variable in the Portal or the mirror); ED's public EDFacts participation files start with SY 2012-13 (ED "EDFacts Data Files" page, archived 2022-01-19) |
| Achievement levels | Below Basic, Basic, Proficient, Advanced | Varies by state |
| Grade levels | Grades 3-8, high school (varies) | Portal years 2009–2018, 2020 (no 2019) |
### Graduation Data
| Data Element | Description | Available Years |
|--------------|-------------|-----------------|
| 4-year ACGR | Adjusted Cohort Graduation Rate | Portal years 2010–2019 |
| Extended-year ACGR (e.g., 5- or 6-year) | Optional and State-adopted; not present in the Portal grad-rates files | — |
| Diploma types | Regular diploma; from 2017-18 also a State-defined alternate diploma for students with the most significant cognitive disabilities | 2017-18 onward |
### Key Identifiers
> **Portal Data Types:** All identifiers are **Int64** in the Portal parquet files. The NCES source format (zero-padded strings) is shown for reference only. When joining with other Portal datasets, join on the integer columns directly.
| ID | Portal Type | NCES Source Format | Level | Example (Int64) |
|----|-------------|-------------------|-------|-----------------|
| `ncessch` | Int64 | 12-char zero-padded | School | `10000500870` |
| `ncessch_num` | Int64 | Same as ncessch | School | `10000500870` |
| `leaid` | Int64 | 7-char zero-padded | District/LEA | `100005` |
| `leaid_num` | Int64 | Same as leaid | District/LEA | `100005` |
| `fips` | Int64 | 2-digit | State | `1` (Alabama) |
### Data Levels
| Level | Identifier | Dataset Path Pattern |
|-------|------------|---------------------|
| School | `ncessch` (Int64) | `edfacts/schools_edfacts_*` |
| District/LEA | `leaid` (Int64) | `edfacts/districts_edfacts_*` |
| State | `fips` (Int64) | Aggregate from lower levels |
### Subgroups Reported
> **Note:** Not all subgroup columns are present in every dataset. Grad rates data does NOT have `sex`, `migrant`, or `military_connected` columns.
| Subgroup | NCES Code | Portal Integer | Column | Available In |
|----------|-----------|----------------|--------|--------------|
| All students | `ALL` | `99` | race, sex, lep, disability | Assessments, Grad Rates |
| Economically disadvantaged | `ECODIS` | `1` | econ_disadvantaged | Assessments, Grad Rates |
| Students with disabilities | `CWD` | `1` | disability | Assessments, Grad Rates |
| English learners | `LEP` | `1` | lep | Assessments, Grad Rates |
| Homeless | `HOM` | `1` | homeless | Assessments, Grad Rates |
| Foster care | `FCS` | `1` | foster_care | Assessments, Grad Rates |
| Migrant | `MIG` | `1` | migrant | Assessments only |
| Military connected | `MIL` | `1` | military_connected | Assessments only |
| Race/ethnicity | Multiple | `1-5, 7, 99` (no `6`; see Race Codes) | race | Assessments, Grad Rates |
| Sex | `M/F` | `1, 2, 99` | sex | Assessments only |
**EDFacts Filter Column Pattern:**
- Special population columns (lep, disability, homeless, etc.) use `1` = subgroup, `99` = total
- Race column uses integer codes (1=White, 2=Black, etc.)
- Sex column uses `1` = Male, `2` = Female, `99` = Total (assessments only)
### Grade Codes (grade_edfacts)
| Code | Grade Level |
|------|-------------|
| `3`-`8` | Grades 3-8 (individual) |
| `9` | Grades 9-12 combined |
| `99` | Total (all grades) |
### Race Codes
> **Empirically verified** from 2018 school assessment data, and confirmed in all 42 mirror files with `race` (assessments and grad rates, every year; pinned-mirror probe 2026-10-30). Only these values appear in the `race` column:
| Code | Category |
|------|----------|
| `1` | White |
| `2` | Black |
| `3` | Hispanic |
| `4` | Asian |
| `5` | American Indian/Alaska Native |
| `7` | Two or More Races |
| `99` | Total |
> **Note:** Code `6` (Native Hawaiian/Pacific Islander) is NOT observed in the data. Codes `8` (Nonresident alien), `9` (Unknown), `20` (Other), `-1`, `-2`, `-3` are also not observed in the race column. These codes may exist in other Portal sources but are absent from EDFacts.
### Sex Codes
| Code | Category |
|------|----------|
| `1` | Male |
| `2` | Female |
| `9` | Unknown |
| `99` | Total |
### Disability Codes
> **Empirically verified** from 2018 school assessment and 2019 grad rate data. Only `1` and `99` are observed in the `disability` column. The expanded codes (0-4) documented in other Portal sources are NOT present in EDFacts datasets.
| Code | Category |
|------|----------|
| `1` | Students with disabilities (IDEA-eligible) |
| `99` | Total (all students) |
### LEP Codes
| Code | Category |
|------|----------|
| `1` | Students who are limited English proficient |
| `99` | All students (total) |
### Special Population Columns
For `homeless`, `migrant`, `econ_disadvantaged`, `foster_care`, `military_connected`:
| Code | Category |
|------|----------|
| `1` | Yes (in subgroup) |
| `99` | Total (all students) |
### Missing Data Codes
| Code | Meaning | When Used |
|------|---------|-----------|
| `-1` | Missing/not reported | Data not reported |
| `-2` | Not applicable | Item doesn't apply to this entity |
| `-3` | Suppressed for privacy | Data suppressed for small N-size |
| `-9` | Not found in Portal data | Not in the Portal varlist corpus (explorer `variable-codes.md`) and in no pinned-mirror school assessment (2009-2018, 2020) or grad-rate (2010-2019) value column (probed 2026-10-28; district files not probed); filters that also drop `-9` are harmless |
| Range values | Exact value suppressed | Range provided instead of exact value |
| `_midpt` suffix | Calculated midpoint of suppressed range | Use for analysis when exact values are suppressed |
**Always use `_midpt` variables for analysis when exact values are suppressed.**
## Data Access
All EDFacts data is fetched via the **Education Data Portal mirror system**. There is no API access.
**Key references:**
- **`mirrors.yaml`** -- Mirror definitions, URL templates, read strategies
- **`datasets-reference.md`** -- Canonical dataset paths (one path works for all mirrors)
- **`fetch-patterns.md`** -- `fetch_from_mirrors()` and `fetch_yearly_from_mirrors()` patterns
> **Truth Hierarchy:** When interpreting variable values, apply this priority:
> 1. **Actual data file** (what you observe in the parquet/CSV) — this IS the truth
> 2. **Live codebook** (.xls in mirror) — authoritative documentation, may lag
> 3. **This skill documentation** — convenient summary, may drift from codebook
>
> If this documentation contradicts the codebook, trust the codebook. If the codebook contradicts observed data, trust the data and investigate.
### Key Datasets
| Dataset | Path | Type | Columns |
|---------|------|------|---------|
| School Assessments | `edfacts/schools_edfacts_assessments_{year}` | Yearly (2009-2018, 2020) | 26 cols |
| School Grad Rates | `edfacts/schools_edfacts_grad_rates_{year}` | Yearly (2010-2019) | 18 cols |
| District Assessments | `edfacts/districts_edfacts_assessments_{year}` | Yearly (2009-2018, 2020) | 23 cols |
| District Grad Rates | `edfacts/districts_edfacts_grad_rates_{year}` | Yearly (2010-2019) | 15 cols |
> **Note:** 2019 assessment data (Portal year 2019 = SY 2019-20) is NOT available (at any level) due to COVID testing waivers. Portal year 2020 is SY 2020-21.
### Codebooks
Codebook `.xls` files are available for both assessment and graduation rate datasets. Use `get_codebook_url()` from `fetch-patterns.md`:
```python
# Assessment codebooks:
url = get_codebook_url("edfacts/codebook_schools_edfacts_assessments")
url = get_codebook_url("edfacts/codebook_districts_edfacts_assessments")
# Graduation rate codebooks:
url = get_codebook_url("edfacts/codebook_schools_edfacts_graduation")
url = get_codebook_url("edfacts/codebook_districts_edfacts_graduation")
```
```r
# get_codebook_url() is a Python helper; in R, build the URL from the mirror root.
# Mirror failover: see `education-data-query/references/fetch-patterns.md` (R pattern)
mirror <- yaml::read_yaml("mirrors.yaml")$mirrors[[1]]
# Assessment codebooks:
url <- paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "edfacts/codebook_schools_edfacts_assessments", ".xls")
url <- paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "edfacts/codebook_districts_edfacts_assessments", ".xls")
# Graduation rate codebooks:
url <- paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "edfacts/codebook_schools_edfacts_graduation", ".xls")
url <- paste0(mirror$root_url, "/", mirror$vintage$hf_revision, "/", "edfacts/codebook_districts_edfacts_graduation", ".xls")
```
> **Codebook naming note:** Graduation rate codebooks use `_graduation` (not `_grad_rates`), while the data files use `_grad_rates`. This follows the same pattern as other Portal sources where codebook names differ from data file names. See `datasets-reference.md` for the authoritative path mapping.
### Dataset Column Differences
Assessment and graduation rate datasets have **different column sets**:
| Column | Assessments | Grad Rates |
|--------|-------------|------------|
| `sex` | Yes (1, 2, 99) | **No** |
| `migrant` | Yes (1, 99) | **No** |
| `military_connected` | Yes (1, 99) | **No** |
| `grade_edfacts` | Yes (3-9, 99) | **No** |
| `read_test_*` / `math_test_*` | Yes | **No** |
| `grad_rate_*` | **No** | Yes |
| `cohort_num` | **No** | Yes |
| `school_name` / `lea_name` | Yes | Yes |
### Filtering
```python
# Grade filtering: grade_edfacts uses integer codes
df = df.filter(pl.col("grade_edfacts") == 4) # Grade 4
df = df.filter(pl.col("grade_edfacts") == 99) # All grades combined
# Subgroup filtering: special population columns use 1/99 pattern
df_total = df.filter(pl.col("sex") == 99) # All students (total)
df_econ = df.filter(pl.col("econ_disadvantaged") == 1) # Economically disadvantaged only
# Race filtering: integer codes
df_black = df.filter(pl.col("race") == 2) # Black students
```
```r
library(dplyr)
# Grade filtering: grade_edfacts uses integer codes
df <- df |> filter(grade_edfacts == 4) # Grade 4
df <- df |> filter(grade_edfacts == 99) # All grades combined
# Subgroup filtering: special population columns use 1/99 pattern
df_total <- df |> filter(sex == 99) # All students (total)
df_econ <- df |> filter(econ_disadvantaged == 1) # Economically disadvantaged only
# Race filtering: integer codes
df_black <- df |> filter(race == 2) # Black students
```
## Common Pitfalls
| Pitfall | Issue | Solution |
|---------|-------|----------|
| Ranking states by proficiency | Different tests, different cut scores make comparisons meaningless | Use NAEP for cross-state comparisons |
| Comparing 2019-20 to other years | COVID testing waivers created data gaps | Note data gap, exclude year |
| Ignoring suppression | Results biased toward larger schools/subgroups | Document suppression rates, use `_midpt` variables |
| Assuming proficiency = same thing | State definitions of "proficient" vary widely | Clarify each state's definition |
| Pre/post ESSA comparison | Different accountability systems (NCLB vs ESSA) | Note policy change at 2015 boundary |
| Using string codes for filtering | Portal uses integer encoding, not NCES strings | Always check actual data values; see encoding tables above |
## Key Policy Context
| Law | Years | Key Features |
|-----|-------|--------------|
| NCLB | 2002-2015 | AYP, 100% proficiency goal, HQT |
| ESSA | 2015-present | State flexibility, multiple indicators |
- **AYP (Adequate Yearly Progress)**: NCLB requirement eliminated by ESSA
- **ESSA Accountability**: States design own systems with federal guardrails
- **N-size**: Minimum students required for reporting (varies by state, typically 10-30)
## CRITICAL WARNING: Cross-State Comparisons
**State assessment proficiency rates CANNOT be compared across states.**
| Factor | Why It Varies |
|--------|---------------|
| Assessment content | Each state creates its own tests |
| Proficiency cut scores | Each state sets own thresholds |
| Standards alignment | State academic standards differ |
| Test difficulty | Not calibrated nationally |
A student "proficient" in one state may score "below basic" in another state taking a harder test with higher cut scores. **Rankings of states by proficiency rates are meaningless.**
Use NAEP (National Assessment of Educational Progress) for valid cross-state comparisons.
### Valid vs. Invalid Analysis Examples
**Valid Analysis:**
```python
# Within-state trend analysis
state_df = df.filter(pl.col("fips") == 6) # California only
trend = state_df.group_by("year").agg(
pl.col("read_test_pct_prof_midpt").mean()
)
# Valid: Same state, same test system
```
```r
library(dplyr)
# Within-state trend analysis
state_df <- df |> filter(fips == 6) # California only
trend <- state_df |>
group_by(year) |>
summarise(read_test_pct_prof_midpt = mean(read_test_pct_prof_midpt, na.rm = TRUE))
# Valid: Same state, same test system
```
**INVALID Analysis:**
```python
# DO NOT DO THIS - Cross-state comparison
# This comparison is MEANINGLESS
state_comparison = df.group_by("fips").agg(
pl.col("read_test_pct_prof_midpt").mean()
).sort("read_test_pct_prof_midpt", descending=True)
# INVALID: Different tests, different standards
```
```r
library(dplyr)
# DO NOT DO THIS - Cross-state comparison
# This comparison is MEANINGLESS
state_comparison <- df |>
group_by(fips) |>
summarise(read_test_pct_prof_midpt = mean(read_test_pct_prof_midpt, na.rm = TRUE)) |>
arrange(desc(read_test_pct_prof_midpt))
# INVALID: Different tests, different standards
```
## Related Data Sources
| Source | Relationship | When to Use |
|--------|--------------|-------------|
| `education-data-source-ccd` | CCD provides school/district demographics | Combining outcome data with school characteristics |
| `education-data-source-crdc` | CRDC has discipline, AP, school climate data | Analyzing school equity alongside achievement |
| `education-data-source-saipe` | SAIPE provides district poverty estimates | Linking poverty to achievement |
| `education-data-source-meps` | MEPS provides school poverty estimates | School-level poverty and assessment analysis |
| `education-data-explorer` | Parent discovery skill | Routing questions to mirror dataset files |
| `education-data-query` | Data fetching | Downloading via mirrors |
## Topic Index
| Topic | Reference File |
|-------|---------------|
| NCLB to ESSA transition | `./references/accountability-context.md` |
| State accountability systems | `./references/accountability-context.md` |
| Federal reporting requirements | `./references/accountability-context.md` |
| Proficiency levels | `./references/assessment-data.md` |
| Why states can't be compared | `./references/assessment-data.md` |
| NAEP comparison | `./references/assessment-data.md` |
| Assessment system changes | `./references/assessment-data.md` |
| ACGR calculation | `./references/graduation-rates.md` |
| Cohort adjustments | `./references/graduation-rates.md` |
| Extended graduation rates | `./references/graduation-rates.md` |
| Diploma types | `./references/graduation-rates.md` |
| Suppression codes | `./references/variable-definitions.md` |
| Missing data values | `./references/variable-definitions.md` |
| Range/midpoint variables | `./references/variable-definitions.md` |
| Participation rates (not in the Portal; use ED's own files) | `./references/variable-definitions.md` |
| COVID-19 data gaps | `./references/data-quality.md` |
| State reporting variations | `./references/data-quality.md` |
| Known data issues | `./references/data-quality.md` |
| Time series breaks | `./references/data-quality.md` |
| Students with disabilities | `./references/subgroup-reporting.md` |
| English learners | `./references/subgroup-reporting.md` |
| Economically disadvantaged | `./references/subgroup-reporting.md` |
| Race/ethnicity reporting | `./references/subgroup-reporting.md` |
| Homeless/foster/migrant | `./references/subgroup-reporting.md` |
| N-size requirements | `./references/subgroup-reporting.md` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |