DAAF-Contribution-Community/daaf / .claude/skills/education-data-query
education-data-query
VerifiedDownloads education datasets from configured mirror sources (parquet/CSV) with local Polars filtering, including versioned mirror vintages (revision-pinned for reproducibility). Use when writing fetch scripts, retrieving CCD, IPEDS, CRDC, SAIPE data, or pinning a fetch to a specific mirror vintage. Load after education-data-explorer — retrieval here, not discovery.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 20.4 KB |
| references/datasets-reference.md | markdown | 23.0 KB |
| references/fetch-patterns.md | markdown | 64.3 KB |
| references/filters-reference.md | markdown | 13.5 KB |
| references/mirrors.yaml | config | 11.2 KB |
| references/query-patterns.md | markdown | 16.4 KB |
| references/vintage-drift.md | markdown | 13.8 KB |
references/mirrors.yaml
View on GitHub# Mirror Configuration for Education Data Portal
#
# This file is the SINGLE SOURCE OF TRUTH for mirror configuration.
# All other documentation references mirrors generically via this file.
#
# ============================================================================
# UNIFIED PATH MODEL
# ============================================================================
# All mirrors use the same canonical path from datasets-reference.md.
# Each mirror's url_template resolves {path} (and, for versioned mirrors,
# {revision}) to a full URL:
#
# url = mirror["url_template"].format(root_url=mirror["root_url"],
# revision=mirror_revision(mirror),
# path=canonical_path,
# format=mirror["format"])
#
# The canonical path is mirror-agnostic: "{source}/{filename}"
# Example: "ccd/schools_ccd_directory" → each mirror appends its own extension.
# {revision} is unused by mirrors that omit it (str.format ignores extra kwargs),
# so the same call works for both versioned and unversioned mirrors.
#
# ============================================================================
# MIRROR VERSIONING: VINTAGE + REVISION PINNING (WHY)
# ============================================================================
# A "vintage" is a dated snapshot of the Education Data Portal captured into one
# Hugging Face dataset repo. A "revision" is the HF git ref (branch, tag, or
# commit SHA) that pins requests to one immutable state of that repo — the URL
# form is `.../resolve/{revision}/{path}` for data and
# `.../tree/{revision}?recursive=true` for discovery (per HF's documented API).
#
# Why pin? The Portal revises historical values between releases WITHOUT schema
# changes — e.g., Portal 0.26.1 retroactively corrected IPEDS Graduation Rates
# 150% for 1996-2023. An unpinned mirror (revision "main") can therefore serve
# silently different numbers to a rerun of the same script. Pinning `hf_revision`
# to an immutable commit SHA makes a research project's fetches byte-reproducible.
#
# Single-value pinning: `vintage.hf_revision` is the ONE place the revision is
# set; url_template and discovery.url_template both consume it. The v2 mirror's
# SHA is now pinned there (see `vintage.hf_revision` below), so every URL (data +
# discovery) pins in lockstep off that one value. Future vintages (v3+) follow the
# same pattern: upload the snapshot, then set its commit SHA in `hf_revision`.
#
# mirror_revision(mirror) returns mirror["vintage"]["hf_revision"] (or "main"
# when a mirror has no vintage block). See fetch-patterns.md for the tested
# Python/R helpers.
#
# Mirrors are tried in priority order (first entry = highest priority).
# The first mirror that successfully serves the requested file is used.
# If a mirror returns 404 or times out, the next mirror is tried.
#
# ============================================================================
# HOW TO ADD A NEW MIRROR
# ============================================================================
# 1. Add a new entry to the `mirrors` list below (position = priority)
# 2. Define url_template using {root_url}, {revision}, {path}, and {format}
# placeholders
# - {path} comes from the single `path` column in datasets-reference.md
# - {format} comes from the mirror's `format` field
# - {revision} comes from the mirror's `vintage.hf_revision` (versioned
# mirrors only; omit both the placeholder and the vintage block otherwise)
# 3. Set read_strategy to match your file format (see options below)
# 4. Define a discovery section so agents can check file availability
# 5. No other documentation files need to change — paths are in datasets-reference.md
# 6. (Optional) Add a metadata section if the mirror serves metadata files
# (e.g., codebook .xls files). See existing mirrors for the format.
# Metadata files are for human reference — not parsed programmatically.
#
# read_strategy options:
# eager_parquet - pl.read_parquet(url) — for columnar formats with embedded schema
# lazy_csv - pl.scan_csv(url, infer_schema_length=10000) — for large text files
#
# discovery.method options:
# http_json - GET a URL that returns JSON listing available files
#
# Discovery is inventory evidence for one surface at one observation time. Never
# encode a mirror as universally complete: endpoint declarations, live API rows,
# Urban bulk objects, and derivative-mirror objects can diverge.
# ============================================================================
mirrors:
- name: huggingface
# root_url ends at /resolve; the {revision} segment is supplied by
# vintage.hf_revision so the whole mirror pins from one value (see the
# MIRROR VERSIONING header above).
root_url: "https://huggingface.co/datasets/brhkim/education_data_portal_mirror_2026q3/resolve"
format: parquet
timeout: 300
url_template: "{root_url}/{revision}/{path}.parquet"
read_strategy: eager_parquet
vintage:
# This mirror snapshots Education Data Portal v0.26.1 (released 2026-08-05).
# Citations of data fetched here MUST use Portal version 0.26.1.
portal_version: "0.26.1"
collected: "2026-08-06" # v2 build validated 2026-08-06 (497 objects; see coverage_notes)
hf_revision: "0ad00ce0e232c96b0642459e4e7326607a8d26aa" # immutable commit SHA of the v2 upload (pinned 2026-08-07)
predecessor:
# The v1 mirror: Portal v0.24.0, frozen for reproducibility of pre-2026q3
# analyses. Not deleted — a rerun that must reproduce old numbers points
# here (see education-data-query/SKILL.md "Mirror Versioning & Reproducibility").
repo: "brhkim/education_data_portal_mirror"
root_url: "https://huggingface.co/datasets/brhkim/education_data_portal_mirror/resolve"
portal_version: "0.24.0"
collected: "2026-02-07"
status: frozen
# Pinned to an immutable commit SHA for byte-exact reproduction. This SHA
# was recorded 2026-08-07 from the HF API after freeze validation (README
# freeze banner present + 5/5 resolve probes returned HTTP 200).
hf_revision: "10fea9c7671e8e0ab90a7976674913bc2d16f8b1"
discovery:
method: http_json
# url_template consumes the same {revision} as the data URLs (tree API form).
url_template: "https://huggingface.co/api/datasets/brhkim/education_data_portal_mirror_2026q3/tree/{revision}?recursive=true"
file_path_key: "path"
file_filter: "*.parquet"
coverage_notes: >
Derivative Parquet/XLS inventory; do not treat it as universally complete or
synchronized with the live API or Urban bulk objects. The v2 build (Portal
v0.26.1) was validated on 2026-08-06: 497 objects (406 Parquet data + 91 XLS
codebooks), plus an in-repo build_manifest.parquet recording per-file
provenance. This closes the eight previously-absent Urban data objects — the
build reports zero Urban-catalog gaps. The tree is size/sha256-verified and
all 406 parquets open with rows>0. Still reconcile dynamically: this is
inventory evidence for one build, not a permanent completeness guarantee.
Parquet format: columnar, compressed (3-10x smaller than CSV), schema embedded.
Provenance: files are Polars-written. Some declare string columns as Arrow
'string_view' in the parquet-native schema (75 of the 406 data parquets at
rev 0ad00ce0, probed 2026-10-28, all carried forward from v1, e.g.
edfacts/{schools,districts}_edfacts_grad_rates_2010..2019;
saipe/districts_saipe, rebuilt for v2, has none). Python (Polars/pyarrow) reads
these fine, but the R arrow binding fails at Table->data.frame with
'cannot handle Array of type <utf8_view>'. R readers MUST use the view-safe
parquet read pattern (read as Arrow Table, cast view types to materialized,
then convert; a no-op on plain-string files). See fetch-patterns.md. Do NOT
use open_dataset()|>collect() as a workaround — it hits the same error.
R large-file timeout: arrow::read_parquet(url) and readr::read_csv(url) both
transfer via R's download.file(), which caps the whole transfer at
getOption("timeout") (default 60s), so large files here (e.g.
ccd/schools_ccd_directory.parquet, ~269MB) truncate at ~60s and silently fall
through to the next mirror in priority order. The timeout: 300 above is not consulted by the R
readers — raise it in-script with options(timeout = max(600, getOption("timeout")))
before the mirror loop (or use curl::multi_download for very large files).
See fetch-patterns.md. Python (Polars/pyarrow) is unaffected.
metadata:
url_template: "{root_url}/{revision}/{path}.{format}"
formats: [xls]
filename_pattern: "codebook_{entity}_{source}_{topic}"
notes: >
Codebook .xls files are co-located with data files in the same directory structure.
Path matches the codebook column in datasets-reference.md with .xls extension.
Codebook URLs pin to the same vintage.hf_revision as the data URLs.
- name: urban_csv
root_url: "https://educationdata.urban.org/csv"
format: csv
timeout: 300
url_template: "{root_url}/{path}.{format}"
read_strategy: lazy_csv
discovery:
method: http_json
url: "https://educationdata.urban.org/api/v1/api-downloads"
# Response is a JSON array of objects with fields:
# id, endpoint_id, sort_order, file_dir, file_name, file_label, file_size, hide
# file_name includes the .csv/.xls extension already. Discovery must deduplicate
# exact "{file_dir}/{file_name}" identities, then return extensionless canonical
# keys: strip one terminal .csv for data or one terminal .xls for codebooks.
# Retain the raw rows separately when endpoint-to-object bridge fidelity matters.
file_dir_key: "file_dir"
file_name_key: "file_name"
file_filter: "*.csv"
coverage_notes: >
Urban-hosted bulk CSV/XLS inventory; do not equate it with endpoint-catalog
declarations or actual live API responses. Dated reconciliation on 2026-07-21
found 958 presentation rows that deduplicated by exact file_dir/file_name to
495 unique objects (404 CSV data files, 91 XLS codebooks). Re-observed
2026-10-28: 990 rows deduplicating to 527 objects (435 CSV, 92 XLS), more
CSV data objects than the pinned HF mirror's 406 data parquets, so a fetch
that falls through to this mirror can reach objects the pinned vintage lacks
(which objects are new was not computed). Manifest rows are
endpoint-linked presentation records and can map many-to-many to objects.
Large files (500MB+); always use lazy loading with filters.
Some datasets are split into yearly files (e.g., ccd/schools_ccd_enrollment_{year}.csv)
while others contain all years in a single file (e.g., fsa/colleges_fsa_grants.csv).
Check the api-downloads manifest to determine which pattern applies.
metadata:
url_template: "{root_url}/{path}.{format}"
formats: [xls]
filename_pattern: "codebook_{entity}_{source}_{topic}"
notes: >
Codebook .xls files use the same {source}/{filename} path structure as data files.
Path from datasets-reference.md codebook column resolves directly.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |