DAAF-Contribution-Community/daaf / .claude/skills/education-data-source-edfacts
education-data-source-edfacts
VerifiedEDFacts — K-12 outcomes: assessment proficiency, ACGR graduation rates, ESSA accountability at school/district level (2009-2020). Within-state trends and subgroup gaps. Complements CCD with outcome data. Cannot compare across states — use NAEP.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 21.1 KB |
| references/accountability-context.md | markdown | 9.6 KB |
| references/assessment-data.md | markdown | 15.5 KB |
| references/data-quality.md | markdown | 17.7 KB |
| references/graduation-rates.md | markdown | 18.4 KB |
| references/subgroup-reporting.md | markdown | 21.1 KB |
| references/variable-definitions.md | markdown | 23.8 KB |
references/data-quality.md
View on GitHub# Data Quality
Known data quality issues, state reporting variations, COVID-19 impacts, and guidance for handling problematic data in EDFacts.
## Contents
- [COVID-19 Data Gaps](#covid-19-data-gaps)
- [State Reporting Variations](#state-reporting-variations)
- [Known Data Issues](#known-data-issues)
- [Time Series Breaks](#time-series-breaks)
- [Suppression Patterns](#suppression-patterns)
- [Data Validation](#data-validation)
- [Documentation Practices](#documentation-practices)
## COVID-19 Data Gaps
### 2019-20 Testing Waivers
**Critical**: ED waived the requirement to administer all required assessments in school year 2019–2020 for each of the 50 States, the District of Columbia, Puerto Rico and the Bureau of Indian Education (Notice of Waivers Granted Under Section 8401 of the ESEA, 85 FR 29935, May 19, 2020; announced March 20, 2020):
| Impact | Details |
|--------|---------|
| Assessment data | No Portal assessment file for Portal year 2019 (SY 2019-20) |
| Participation rates | Not meaningful for 2019-20 |
| Federal waivers | All 50 States, DC, Puerto Rico and the BIE received 2019-20 assessment waivers (85 FR 29935) |
### What's Available for 2019-20
| Data Type | Availability |
|-----------|--------------|
| Assessment proficiency | **Absent from the Portal** (no Portal year 2019 assessment file) |
| Graduation rates | Mostly available (cohort-based) |
| Directory data | Available |
| Enrollment data | Available |
### 2020-21 Partial Return
The 2020-21 school year had mixed testing:
| Variation | Impact |
|-----------|--------|
| State decisions | Some states tested, some didn't |
| Participation rates | Often below 95% (unsourced here; participation rates are in ED's own files, not the Portal) |
| Comparability | Limited comparability to prior years |
### 2021-22 Onward
Testing largely resumed, but considerations remain (these years are not in the Portal assessment files, which end at Portal year 2020 = SY 2020-21):
| Issue | Guidance |
|-------|----------|
| Learning loss | Score drops reflect COVID impact |
| Changed baselines | Not directly comparable to pre-COVID |
| Ongoing effects | Recovery still in progress |
### Handling COVID Years in Analysis
```python
def exclude_covid_years(df, assessment_col):
"""Exclude COVID-affected years from trend analysis."""
# Portal assessment `year` = fall of the school year.
# 2019 = SY 2019-20 (spring 2020 testing waived; no Portal assessment file)
# 2020 = SY 2020-21 (partial testing); the last Portal assessment year
return df.filter(
~pl.col("year").is_in([2019, 2020])
)
def flag_covid_years(df):
"""Add flag for COVID-affected years."""
return df.with_columns(
pl.when(pl.col("year") == 2019)
.then(pl.lit("no_testing"))
.when(pl.col("year") == 2020)
.then(pl.lit("partial_testing"))
.otherwise(pl.lit("normal"))
.alias("covid_status")
)
```
```r
library(dplyr)
# Exclude COVID-affected years from trend analysis
# Portal assessment `year` = fall of the school year:
# 2019 = SY 2019-20 (spring 2020 testing waived; no Portal assessment file)
# 2020 = SY 2020-21 (partial testing); the last Portal assessment year
df_no_covid <- df |> filter(!year %in% c(2019, 2020))
# Add flag for COVID-affected years
df <- df |>
mutate(covid_status = case_when(
year == 2019 ~ "no_testing",
year == 2020 ~ "partial_testing",
TRUE ~ "normal"
))
```
### Reporting COVID Gaps
When presenting findings:
> "Data for 2019-20 is largely unavailable due to federal assessment waivers during COVID-19. Data for 2020-21 reflects partial testing with participation below typical levels. Trends should be interpreted with caution around these years."
## State Reporting Variations
### Timing of Data Submission
States submit data on different schedules:
| Variation | Impact |
|-----------|--------|
| Submission deadlines | Data completeness varies |
| Resubmission | Some states update data |
| Lag time | 1-2 years from collection to availability |
### Definition Variations
Despite federal requirements, states interpret definitions differently:
| Concept | How It Varies |
|---------|---------------|
| Economically disadvantaged | FRPL, CEP, other measures |
| School levels | Grade spans differ |
| Charter classification | Reporting practices differ |
| Transfer documentation | Tracking rigor varies |
### Economically Disadvantaged Definition
States define "economically disadvantaged" differently:
| Definition | States Using |
|------------|--------------|
| Free lunch eligible | Some states |
| Free or reduced lunch | Many states |
| Direct certification | Increasing |
| CEP schools (100%) | Growing issue |
### Community Eligibility Provision (CEP) Impact
CEP schools serve all students free meals regardless of income:
| Issue | Impact |
|-------|--------|
| No individual applications | Can't identify FRPL-eligible |
| Often marked 100% | Inflates economically disadvantaged counts |
| Comparability | Pre/post CEP not comparable |
```python
# Flag potential CEP schools
def flag_potential_cep(df):
"""Flag schools that might be CEP (100% economically disadvantaged)."""
return df.with_columns(
(pl.col("econ_disadvantaged_pct") >= 99).alias("likely_cep")
)
```
```r
library(dplyr)
# Flag potential CEP schools
df <- df |> mutate(likely_cep = econ_disadvantaged_pct >= 99)
```
## Known Data Issues
### State-Specific Issues
Documented data quality issues by state (examples):
| State | Year | Issue |
|-------|------|-------|
| Virginia | 2016-17 | Grade 5-8 math proficiency shares inaccurate |
| Various | 2019-20 | COVID testing waivers |
| Various | Multiple | Assessment system transitions |
### Checking for Known Issues
1. **Review state documentation**: Check state education agency websites
2. **Look for unusual patterns**: Large year-over-year changes
3. **Verify with multiple sources**: Cross-reference with state data
4. **Contact state if needed**: State data offices can clarify
### Common Data Anomalies
| Anomaly | Possible Cause |
|---------|----------------|
| Large proficiency jump | Assessment change |
| Large proficiency drop | Higher standards/new test |
| Missing year | Data collection issue |
| 100% proficiency | Very small school or data error |
| 0% proficiency | Very small school or data error |
## Time Series Breaks
### Assessment System Changes
States periodically adopt new assessments:
| Period | Major Changes |
|--------|---------------|
| 2014-2016 | Common Core adoption |
| 2015-2017 | PARCC/SBAC implementation |
| 2017-2018 | ESSA transition |
| Ongoing | State-specific changes |
### Detecting Time Series Breaks
```python
def detect_breaks(df, state_fips, variable, threshold=10):
"""
Detect potential time series breaks.
Parameters:
-----------
df : DataFrame with year and variable columns
state_fips : State to analyze
variable : Variable to check for breaks
threshold : Percentage point change to flag (default 10)
Returns:
--------
DataFrame with flagged years
"""
state_data = (df
.filter(pl.col("fips") == state_fips)
.sort("year")
)
# Calculate year-over-year change
state_data = state_data.with_columns(
(pl.col(variable) - pl.col(variable).shift(1)).alias("yoy_change")
)
# Flag large changes
breaks = state_data.filter(
pl.col("yoy_change").abs() > threshold
)
return breaks
```
```r
library(dplyr)
# Detect potential time series breaks
# threshold: percentage point change to flag (default 10)
state_data <- df |>
filter(fips == state_fips) |>
arrange(year) |>
mutate(yoy_change = .data[[variable]] - lag(.data[[variable]]))
# Flag large changes
breaks <- state_data |> filter(abs(yoy_change) > threshold)
```
### Handling Time Series Breaks
| Approach | When to Use |
|----------|-------------|
| Split analysis | Pre and post change separately |
| Exclude transition | Drop year(s) of change |
| Document break | Note limitation in findings |
| Standardize scores | If scale scores available |
### Example: Segmented Analysis
```python
def segmented_trend(df, state_fips, variable, break_year):
"""Analyze trends separately before and after break."""
state_data = df.filter(pl.col("fips") == state_fips)
pre_break = state_data.filter(pl.col("year") < break_year)
post_break = state_data.filter(pl.col("year") >= break_year)
results = {
"pre_break_trend": calculate_trend(pre_break, variable),
"post_break_trend": calculate_trend(post_break, variable),
"break_year": break_year,
"note": "Trends should not be compared across the break"
}
return results
```
```r
library(dplyr)
# Analyze trends separately before and after break
state_data <- df |> filter(fips == state_fips)
pre_break <- state_data |> filter(year < break_year)
post_break <- state_data |> filter(year >= break_year)
# Note: Trends should not be compared across the break
```
## Suppression Patterns
### Who Gets Suppressed
Suppression disproportionately affects:
| Population | Suppression Rate |
|------------|------------------|
| Small schools | High |
| Rural schools | High |
| Small subgroups | Very high |
| Students with disabilities | High |
| English learners | Moderate to high |
| Small racial groups | Very high |
### Suppression Impact on Analysis
| Issue | Impact |
|-------|--------|
| Sample selection | Only larger schools remain |
| Subgroup invisibility | Can't see small group outcomes |
| Bias | Survivors may not be representative |
| Aggregation | May need to aggregate up |
### Analyzing Suppression Patterns
```python
def suppression_analysis(df, value_col, group_col=None):
"""Analyze suppression patterns."""
if group_col:
analysis = (df
.group_by(group_col)
.agg([
pl.len().alias("total"),
(pl.col(value_col) == -3).sum().alias("suppressed"),
(pl.col(value_col) >= 0).sum().alias("valid")
])
.with_columns(
(pl.col("suppressed") / pl.col("total") * 100).alias("pct_suppressed")
)
.sort("pct_suppressed", descending=True)
)
else:
total = len(df)
suppressed = (df[value_col] == -3).sum()
analysis = {
"total": total,
"suppressed": suppressed,
"pct_suppressed": suppressed / total * 100
}
return analysis
```
```r
library(dplyr)
# Analyze suppression patterns by group
analysis <- df |>
group_by(.data[[group_col]]) |>
summarise(
total = n(),
suppressed = sum(.data[[value_col]] == -3, na.rm = TRUE),
valid = sum(.data[[value_col]] >= 0, na.rm = TRUE)
) |>
mutate(pct_suppressed = suppressed / total * 100) |>
arrange(desc(pct_suppressed))
# Or overall (no grouping)
total <- nrow(df)
suppressed <- sum(df[[value_col]] == -3, na.rm = TRUE)
pct_suppressed <- suppressed / total * 100
```
### Mitigating Suppression Bias
| Strategy | Description |
|----------|-------------|
| Aggregate to district | Less suppression at district level |
| Pool years | Combine multiple years |
| Pool subgroups | Broader categories |
| Document limitation | Note excluded records |
| Use multiple imputation | Advanced technique |
## Data Validation
### Basic Validation Checks
```python
def validate_edfacts(df):
"""Basic validation for EDFacts data."""
issues = []
# Check for impossible percentages
pct_cols = [c for c in df.columns if "pct" in c and "_midpt" in c]
for col in pct_cols:
invalid = df.filter(
(pl.col(col) > 100) |
((pl.col(col) < 0) & (~pl.col(col).is_in([-1, -2, -3, -9])))
)
if len(invalid) > 0:
issues.append(f"Invalid percentages in {col}: {len(invalid)} records")
# Check for year coverage
years = sorted(df["year"].unique().to_list())
expected_years = list(range(min(years), max(years) + 1))
missing_years = set(expected_years) - set(years)
if missing_years:
issues.append(f"Missing years: {missing_years}")
# Check for ID consistency
id_cols = ["ncessch", "leaid", "fips"]
for col in id_cols:
if col in df.columns:
nulls = df[col].null_count()
if nulls > 0:
issues.append(f"Null IDs in {col}: {nulls} records")
return issues
```
```r
library(dplyr)
library(stringr)
# Basic validation for EDFacts data
issues <- character()
# Check for impossible percentages
pct_cols <- names(df)[str_detect(names(df), "pct") & str_detect(names(df), "_midpt")]
for (col in pct_cols) {
invalid_n <- df |>
filter(.data[[col]] > 100 | (.data[[col]] < 0 & !.data[[col]] %in% c(-1, -2, -3, -9))) |>
nrow()
if (invalid_n > 0) issues <- c(issues, paste0("Invalid percentages in ", col, ": ", invalid_n, " records"))
}
# Check for year coverage
years <- sort(unique(df$year))
expected_years <- seq(min(years), max(years))
missing_years <- setdiff(expected_years, years)
if (length(missing_years) > 0) issues <- c(issues, paste0("Missing years: ", paste(missing_years, collapse = ", ")))
# Check for ID consistency
for (col in c("ncessch", "leaid", "fips")) {
if (col %in% names(df)) {
nulls <- sum(is.na(df[[col]]))
if (nulls > 0) issues <- c(issues, paste0("Null IDs in ", col, ": ", nulls, " records"))
}
}
cat(issues, sep = "\n")
```
### Cross-Source Validation
Compare EDFacts data with other sources:
| Source | Compare To | Check |
|--------|------------|-------|
| EDFacts enrollment | CCD enrollment | Should be similar |
| EDFacts grad rates | State reports | Should match |
| EDFacts assessments | State reports | Should match |
```python
def cross_validate(edfacts_df, ccd_df, merge_keys, compare_cols):
"""Cross-validate EDFacts with CCD data."""
merged = edfacts_df.join(
ccd_df,
on=merge_keys,
suffix="_ccd"
)
discrepancies = []
for col in compare_cols:
ccd_col = f"{col}_ccd"
diff = merged.with_columns(
(pl.col(col) - pl.col(ccd_col)).abs().alias(f"{col}_diff")
)
large_diff = diff.filter(pl.col(f"{col}_diff") > 5)
if len(large_diff) > 0:
discrepancies.append({
"column": col,
"records_with_large_diff": len(large_diff)
})
return discrepancies
```
```r
library(dplyr)
# Cross-validate EDFacts with CCD data
merged <- edfacts_df |>
left_join(ccd_df, by = merge_keys, suffix = c("", "_ccd"))
for (col in compare_cols) {
ccd_col <- paste0(col, "_ccd")
merged_diff <- merged |>
mutate(diff = abs(.data[[col]] - .data[[ccd_col]])) |>
filter(diff > 5)
if (nrow(merged_diff) > 0) {
cat(paste0("Large discrepancies in ", col, ": ", nrow(merged_diff), " records\n"))
}
}
```
## Documentation Practices
### What to Document
For any EDFacts analysis, document:
| Element | Description |
|---------|-------------|
| Data version | Which release of the data |
| Years included | Time period of analysis |
| Filters applied | States, grades, subgroups |
| Missing data handling | How -1, -2, -3, -9 treated |
| Suppression impact | % records suppressed |
| Known issues | Any data quality issues |
| COVID impact | How COVID years handled |
### Documentation Template
```markdown
## Data Notes
### Source
- Data: EDFacts via Urban Institute Education Data Portal
- Version: [version number]
- Access date: [date]
### Scope
- Years: [range]
- States: [list or "all"]
- Grades: [list]
- Subgroups: [list]
### Data Quality
- Records suppressed (code -3): X%
- Records missing/not reported (code -1): X%
- Records with ranges (not exact): X%
### Limitations
- [COVID-19 impact if relevant]
- [Assessment changes if relevant]
- [Other issues]
### Processing
- Missing values handled by: [approach]
- Midpoint variables used for: [list]
- Exclusions: [any records excluded]
```
### Example: Full Documentation
```markdown
## Data Notes for State Proficiency Analysis
### Source
- Data: EDFacts Assessment Data via Urban Institute Education Data Portal
- Version: 2024.1
- Access date: January 15, 2024
### Scope
- Years: 2017-18 through 2020-21 (Portal years 2017, 2018, 2020; no Portal year 2019 assessment file)
- States: All 50 states + DC
- Grades: Grade 4 reading
- Subgroups: All students, economically disadvantaged, students with disabilities
### Data Quality
- 2019-20 (Portal year 2019) absent from the Portal due to COVID testing waivers
- 2020-21 participation rates are not in the Portal; check ED's own EDFacts participation files before comparing 2020-21 with earlier years
- 12% of school-year-subgroup records suppressed (code -3)
- 3% of records not applicable (code -2)
### Limitations
- Proficiency rates NOT comparable across states
- Several states changed assessments during period (breaks noted in text)
- CEP schools may inflate economically disadvantaged counts
### Processing
- Used _midpt variables for suppressed exact values
- Excluded records with -1, -2, -3 codes
- Weighted state averages by student count
### Citation
EDFacts Assessment Data, Education Data Portal (Version [version]),
Urban Institute, accessed [date],
https://educationdata.urban.org/documentation/,
made available under the ODC Attribution License.
```
## Recommendations Summary
### Before Analysis
1. Check for known data quality issues
2. Identify COVID-affected years
3. Look for assessment system changes
4. Assess suppression rates
### During Analysis
1. Document all data handling decisions
2. Use midpoint variables consistently
3. Flag uncertain estimates
4. Consider suppression bias
### When Reporting
1. Note all data limitations
2. Describe COVID data gaps
3. State that cross-state comparisons are invalid
4. Provide full data documentation
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |