brycewang-stanford/Auto-Empirical-Research-Skills / skills/17-DAAF-Contribution-Community-daaf/dot-claude/skills/education-data-source-edfacts
education-data-source-edfacts
VerifiedEDFacts — K-12 outcomes: assessment proficiency, ACGR graduation rates, ESSA accountability at school/district level (2009-2020). Within-state trends and subgroup gaps. Complements CCD with outcome data. Cannot compare across states — use NAEP.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 18.2 KB |
| references/accountability-context.md | markdown | 9.6 KB |
| references/assessment-data.md | markdown | 12.9 KB |
| references/data-quality.md | markdown | 13.7 KB |
| references/graduation-rates.md | markdown | 13.4 KB |
| references/subgroup-reporting.md | markdown | 17.6 KB |
| references/variable-definitions.md | markdown | 15.2 KB |
references/assessment-data.md
View on GitHub# Assessment Data
Understanding state assessment data in EDFacts, including proficiency levels, why cross-state comparisons are invalid, and proper analysis approaches.
## Contents
- [Critical Limitation: No Cross-State Comparisons](#critical-limitation-no-cross-state-comparisons)
- [Why States Cannot Be Compared](#why-states-cannot-be-compared)
- [NAEP: The Exception](#naep-the-exception)
- [Proficiency Levels](#proficiency-levels)
- [Assessment Data Structure](#assessment-data-structure)
- [Range and Midpoint Variables](#range-and-midpoint-variables)
- [Time Series Considerations](#time-series-considerations)
- [Valid Analysis Approaches](#valid-analysis-approaches)
## Critical Limitation: No Cross-State Comparisons
**This is the most important caveat for EDFacts assessment data.**
State assessment proficiency rates are **NOT comparable across states**. Rankings or comparisons of states by proficiency percentages are **meaningless** and **misleading**.
### The Core Problem
Each state administers its own assessment with its own proficiency standards:
| Component | State A | State B |
|-----------|---------|---------|
| Test content | State-designed | State-designed |
| Difficulty level | Varies | Varies |
| Cut scores | State-determined | State-determined |
| Proficiency definition | State-specific | State-specific |
A student who scores "proficient" in one state might score "below basic" in another state—not because they learned more, but because one state has lower standards.
## Why States Cannot Be Compared
### 1. Different Assessment Content
Each state develops or selects its own assessment:
- Different question types
- Different content coverage
- Different cognitive demand levels
- Different alignment to standards
### 2. Different Academic Standards
State learning standards vary significantly:
- Grade-level expectations differ
- Content scope and sequence varies
- Depth of knowledge requirements differ
### 3. Different Proficiency Cut Scores
The score needed to be "proficient" varies dramatically:
| Example | Impact |
|---------|--------|
| State with low cut score | Higher proficiency rates |
| State with high cut score | Lower proficiency rates |
| Same student performance | Different labels |
### 4. Political and Economic Pressures
States face pressures that influence cut scores:
- Avoid having too many "failing" schools
- Balance rigor with achievable targets
- Respond to stakeholder concerns
### Evidence: NAEP Mapping Studies
NCES periodically maps state proficiency standards onto the NAEP scale:
| Finding | Implication |
|---------|-------------|
| State cut scores vary by 40+ NAEP points | Huge differences in what "proficient" means |
| Some state "proficient" = NAEP "Basic" | States labeling students proficient at low levels |
| Variation persists over time | Not converging to common standard |
### Example: Mapping Results
From NCES mapping studies (illustrative):
| State | Grade 4 Reading "Proficient" | NAEP Equivalent |
|-------|------------------------------|-----------------|
| High-standards state | 250 scale score | NAEP Proficient |
| Low-standards state | 215 scale score | NAEP Basic |
| Difference | 35 points | One achievement level |
**The same student would be "proficient" in one state and "basic" in another.**
## NAEP: The Exception
The National Assessment of Educational Progress (NAEP) IS comparable across states:
| Feature | NAEP | State Assessments |
|---------|------|-------------------|
| Test content | Identical nationally | State-specific |
| Cut scores | Same everywhere | State-determined |
| Administration | Standardized | Varies |
| Comparability | Cross-state valid | Within-state only |
### NAEP Limitations
| Limitation | Impact |
|------------|--------|
| Sample-based | No individual student scores |
| Limited grades | 4, 8, 12 (not tested every year) |
| Limited subjects | Reading, math, science, writing |
| Not for accountability | Cannot identify individual schools |
### Using NAEP for Cross-State Analysis
For valid cross-state comparisons, use NAEP data (not available in the Portal mirror):
```python
# NAEP data is available from nationsreportcard.gov
# or the NAEP Data Explorer — it is NOT in the Education Data Portal.
#
# Example: Valid cross-state comparison (conceptual — NAEP data is external)
# naep_scores = pl.read_csv("naep_grade4_reading_2022.csv")
# state_rankings = naep_scores.sort("avg_scale_score", descending=True)
# This comparison IS meaningful because NAEP uses the same test nationwide
```
## Proficiency Levels
### Standard Achievement Levels
Most states report four levels (labels vary):
| Level | Common Labels | Description |
|-------|---------------|-------------|
| Level 4 | Advanced, Exceeds | Above grade-level expectations |
| Level 3 | Proficient, Meets | At grade-level expectations |
| Level 2 | Basic, Approaches | Partial mastery |
| Level 1 | Below Basic, Does Not Meet | Minimal mastery |
### Proficiency Rate Definition
**Proficiency rate** typically means % of students at Levels 3 + 4:
```
Proficiency Rate = (# Proficient + # Advanced) / # Tested × 100
```
### Cautions About Proficiency
| Issue | Explanation |
|-------|-------------|
| Binary reduction | Loses information about distribution |
| Cut score proximity | Students just below and just above are similar |
| Incentive to focus on "bubble" | Students near cut score get disproportionate attention |
| Doesn't capture growth | Proficiency status, not progress |
## Assessment Data Structure
### Variables in EDFacts Assessment Data
| Variable | Description |
|----------|-------------|
| `read_test_pct_prof_low` | Reading proficiency %, lower bound |
| `read_test_pct_prof_high` | Reading proficiency %, upper bound |
| `read_test_pct_prof_midpt` | Reading proficiency %, midpoint |
| `math_test_pct_prof_low` | Math proficiency %, lower bound |
| `math_test_pct_prof_high` | Math proficiency %, upper bound |
| `math_test_pct_prof_midpt` | Math proficiency %, midpoint |
| `read_test_num_valid` | Number of valid reading scores |
| `math_test_num_valid` | Number of valid math scores |
### Subject Coverage
| Subject | Availability |
|---------|--------------|
| Reading/ELA | All states, grades 3-8 + high school |
| Mathematics | All states, grades 3-8 + high school |
| Science | Many states, selected grades |
### Grade Levels
Assessment required at:
- Grades 3-8 annually
- Once in high school (typically grade 10 or 11)
## Range and Midpoint Variables
### Why Range Variables Exist
When exact proficiency rates would reveal individual students, data is suppressed and reported as ranges:
| Scenario | Reporting |
|----------|-----------|
| Large school | Exact percentage |
| Small school/subgroup | Range (e.g., 30-40%) |
### Range Variable Structure
| Variable Suffix | Meaning |
|-----------------|---------|
| `_low` | Lower bound of range |
| `_high` | Upper bound of range |
| `_midpt` | Calculated midpoint: (low + high) / 2 |
### Using Midpoint Variables
**Always use `_midpt` variables for quantitative analysis**:
```python
# Correct approach
df.select([
"read_test_pct_prof_midpt", # Use midpoint
"math_test_pct_prof_midpt" # Use midpoint
])
# Avoid
df.select([
"read_test_pct_prof_low", # Only bounds, not useful alone
"read_test_pct_prof_high" # Only bounds, not useful alone
])
```
### Midpoint Caveats
| Caveat | Implication |
|--------|-------------|
| Introduces uncertainty | True value anywhere in range |
| Wider ranges = more uncertainty | Document range widths |
| Not exact | Treat as estimate |
## Time Series Considerations
### Assessment System Changes
States periodically change their assessments:
| Change Type | Impact |
|-------------|--------|
| New test vendor | Break in time series |
| New standards | Break in time series |
| Re-norming | Break in time series |
| New cut scores | Break in time series |
### Major Transition Periods
| Period | Event | Impact |
|--------|-------|--------|
| 2014-2017 | Common Core transition | Many states adopted new tests |
| 2015-2017 | PARCC/SBAC implementation | Consortium assessments |
| 2019-2020 | COVID-19 | Testing largely waived |
| 2020-2021 | Return to testing | Varied state approaches |
### Identifying Assessment Changes
```python
def detect_assessment_changes(df, state_fips, variable):
"""Flag potential assessment system changes."""
state_data = df.filter(pl.col("fips") == state_fips).sort("year")
# Calculate year-over-year change
state_data = state_data.with_columns(
(pl.col(variable) - pl.col(variable).shift(1)).alias("yoy_change")
)
# Large changes (>10 points) suggest assessment change
suspicious = state_data.filter(pl.col("yoy_change").abs() > 10)
return suspicious
```
### Handling Time Series Breaks
| Approach | When to Use |
|----------|-------------|
| Split series | Analyze pre- and post-change separately |
| Exclude transition year | Drop year of change |
| Document break | Note limitation in findings |
| Verify with state | Check state documentation |
## Valid Analysis Approaches
### Valid Comparisons
| Comparison | Validity | Notes |
|------------|----------|-------|
| Same state over time | Valid | If same assessment system |
| Schools within state | Valid | Same test, same standards |
| Districts within state | Valid | Same test, same standards |
| Subgroups within school | Valid | Check suppression |
| Cross-state proficiency | **INVALID** | Different tests and standards |
### Example: Valid Trend Analysis
```python
# Valid: Within-state trend analysis
def analyze_state_trend(df, state_fips, start_year, end_year):
"""Analyze proficiency trends within a single state."""
state_data = (df
.filter(pl.col("fips") == state_fips)
.filter(pl.col("year").is_between(start_year, end_year))
)
# Check for assessment changes in period
# (You would implement detection logic here)
trend = (state_data
.group_by("year")
.agg(pl.col("read_test_pct_prof_midpt").mean())
.sort("year")
)
return trend
```
### Example: Within-State School Comparison
```python
# Valid: Compare schools within same state
def compare_schools_within_state(df, state_fips, year):
"""Compare schools within the same state."""
state_schools = (df
.filter(pl.col("fips") == state_fips)
.filter(pl.col("year") == year)
.select([
"ncessch",
"school_name",
"read_test_pct_prof_midpt",
"math_test_pct_prof_midpt"
])
)
return state_schools.sort("read_test_pct_prof_midpt", descending=True)
```
### Invalid Analysis Example
```python
# INVALID: Cross-state comparison
# DO NOT DO THIS
def invalid_state_ranking(df, year):
"""THIS IS INVALID - DO NOT USE."""
# This ranking is MEANINGLESS
state_ranking = (df
.filter(pl.col("year") == year)
.group_by("fips")
.agg(pl.col("read_test_pct_prof_midpt").mean())
.sort("read_test_pct_prof_midpt", descending=True)
)
# WARNING: This comparison is invalid
# States have different tests and standards
return state_ranking # DO NOT USE
```
## Participation Rates
### 95% Participation Requirement
ESSA requires 95% student participation in assessments:
| Requirement | Details |
|-------------|---------|
| All students | 95% of each school |
| Each subgroup | 95% of each reportable subgroup |
| Consequences | States must address low participation |
### Participation Data Variables
| Variable | Description |
|----------|-------------|
| `read_test_pct_part` | Reading participation rate |
| `math_test_pct_part` | Math participation rate |
### Low Participation Concerns
| Issue | Impact |
|-------|--------|
| Selection bias | Who opts out may differ systematically |
| Proficiency inflation | If low performers opt out, rates inflated |
| Accountability issues | May not meet 95% requirement |
### Opt-Out Movements
Some states/communities have significant opt-out rates:
- Certain states have organized opt-out movements
- Higher income communities sometimes have higher opt-out
- Can significantly affect school-level data
## Recommendations for Analysis
### Always
1. **Clarify scope**: State clearly that analysis is within-state only
2. **Check for breaks**: Identify assessment system changes
3. **Use midpoint variables**: For suppressed data
4. **Document suppression**: Report suppression rates
5. **Note COVID gap**: 2019-20 data largely missing
### Never
1. **Never rank states by proficiency**: Invalid comparison
2. **Never claim cross-state differences**: Tests are different
3. **Never ignore assessment changes**: Creates false trends
4. **Never present proficiency as absolute**: It's state-specific
### Communication Template
When reporting findings:
> "Proficiency rates in [State] increased from X% to Y% between [Year1] and [Year2]. Note that state assessment proficiency rates reflect [State]'s specific standards and cannot be compared to other states. For cross-state comparisons, see NAEP results."
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |