hashgraph-online/awesome-codex-plugins / plugins/alpoxdev/hypercore/skills/hyper-data
hyper-data
VerifiedUse this skill when the user asks a question about a dataset - a CSV, Parquet, JSON, or database export - and wants the answer computed: totals, group comparisons, trends, distributions, joins, or a data-quality check. It probes the installed engines, chooses in-memory or in-place placement from a size and free-memory check, runs the smallest query that answers the question, and saves `.hyper/data/[slug]/report.md` with the answer first, then the query, grain, filters, null and duplicate handling, row counts, timing, and the input digest. Do not use for schema, migration, query-tuning, or index work; GA4 property report data; dashboard UI; or model training and forecasting.
| File | Kind | Size |
|---|---|---|
| SKILL.ko.md | markdown | 17.0 KB |
| SKILL.md | skill md | 15.3 KB |
| assets/evals/hyper-data-cases.jsonl | other | 11.2 KB |
| references/polars-api.ko.md | markdown | 4.6 KB |
| references/polars-api.md | markdown | 4.3 KB |
| rules/charts.ko.md | markdown | 6.2 KB |
| rules/charts.md | markdown | 5.3 KB |
| rules/engine-selection.ko.md | markdown | 7.0 KB |
| rules/engine-selection.md | markdown | 6.1 KB |
| rules/placement.ko.md | markdown | 5.5 KB |
| rules/placement.md | markdown | 4.5 KB |
| rules/query-evidence.ko.md | markdown | 5.9 KB |
| rules/query-evidence.md | markdown | 5.3 KB |
| scripts/probe-engines.shexecutable | script | 8.2 KB |
rules/placement.md
View on GitHub# Placement
Use this rule before the first query on any file larger than a few tens of megabytes, on remote data, or on data that lives in a database. It decides where the computation lives so one question does not re-scan a file ten times or swap the machine.
## 1. Probe Before Computing
Three facts decide placement: file size, free memory, and (when unclear) a row estimate from a streaming scan.
- File size is `os.path.getsize()` or `ls -l`; it is not the working set.
- Free memory comes from the probe's `memory.freeBytes` when present. Physical RAM is not free memory; a machine with 32 GB installed and 2 GB free does not hold a 10 GB working set.
- A row estimate comes from a streaming count, never from loading the file: DuckDB `SELECT count(*) FROM 'data.csv'` streams the scan.
Estimate the working set from the columns the question touches. CSV typically expands 2-5x in memory because string columns dominate; Parquet expands less predictably because compressed columns can inflate 10x when decompressed. A 4 GB Parquet file can be a 40 GB working set.
## 2. Load Once: Within About 25% of Free Memory
This threshold is a package choice, not a vendor guarantee: load into memory only when the working set stays within about 25% of free RAM, and the session will run repeated queries against it.
- Load once and iterate: `CREATE TABLE t AS SELECT ...` in DuckDB, or a collected DataFrame in Polars. One scan up front turns every later query from a file re-scan into milliseconds.
- Prune at load time: select only the columns the question touches and drop obvious dross, so the resident table is the working set, not the raw file.
- The 25% line exists because the OS, the interpreter, and every other process also need memory. A working set at 90% of free RAM swaps; a working set at 25% does not.
## 3. In Place or Streaming: Single-Pass or Larger Than Memory
- A single-pass question queries the file directly (`FROM 'data.csv'`). Loading first is pure waste.
- Larger than RAM, SQL-shaped: cap DuckDB and let it spill.
```sql
SET memory_limit = '4GB';
SET temp_directory = '/tmp/duckdb_spill';
```
Aggregations, sorts, and window functions then run out-of-core. Slower, but bounded.
- Larger than RAM, DataFrame-shaped: Polars streaming on a lazy plan.
```python
out = (pl.scan_parquet("huge.parquet")
.filter(pl.col("status") == "active")
.group_by("region").agg(pl.len())
.collect(engine="streaming"))
```
Keep the plan lazy end to end; an intermediate `collect()` breaks it into eager pieces.
- Hand-rolled chunk loops are the last resort. The engines' own out-of-core paths are faster and simpler than a manual read-N-rows loop.
## 4. Push the Query to the Data
- Remote files over `https` or S3: DuckDB reads Parquet and CSV with projection and predicate pushdown when the storage supports it. Parquet can prune row groups; CSV generally means scanning the stream. Confirm the format and the preinstalled extension first; never claim reduced reads without the engine reporting them.
- Data in a database the user can query: aggregate there at the reader's required grain and return the result rows. A group-by result is kilobytes; the source table is gigabytes.
- Data on another machine the user can execute on: run the query there and return the aggregate. Decision rule: when the result is much smaller than the data, move the query; when local iteration on one slice is planned, move a pruned copy of that slice once and then work locally.
- Never load a dataset larger than free memory fully into memory. Swapping stalls the whole machine; streaming merely takes longer. When the probe says the working set is close to free RAM, choose streaming, not hope.
## Sources
> Checked 2026-11-09.
| Claim | Source |
|---|---|
| Loading once only within about a quarter of free RAM, and streaming or querying in place beyond that | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/SKILL.md`, `references/placement.md` (mechanism reference, re-authored) |
| Moving the query to the data when the result is much smaller than the input | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/references/placement.md` (mechanism reference, re-authored) |
| The in-place SQL and streaming-plan recipes | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/references/placement.md` (mechanism reference, re-authored) |
### Evidence grade
`LOCAL` - the 25% threshold is this package's chosen threshold, labeled as one here. It is the mechanism's heuristic, not a measured optimum for every machine, and the probe's free-memory figure is what the threshold compares against.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |