Skip to content

hashgraph-online/awesome-codex-plugins / plugins/alpoxdev/hypercore/skills/hyper-data

hyper-data

Verified

Use this skill when the user asks a question about a dataset - a CSV, Parquet, JSON, or database export - and wants the answer computed: totals, group comparisons, trends, distributions, joins, or a data-quality check. It probes the installed engines, chooses in-memory or in-place placement from a size and free-memory check, runs the smallest query that answers the question, and saves `.hyper/data/[slug]/report.md` with the answer first, then the query, grain, filters, null and duplicate handling, row counts, timing, and the input digest. Do not use for schema, migration, query-tuning, or index work; GA4 property report data; dashboard UI; or model training and forecasting.

FileKindSize
SKILL.ko.mdmarkdown17.3 KB
SKILL.mdskill md15.6 KB
assets/evals/hyper-data-cases.jsonlother11.2 KB
references/polars-api.ko.mdmarkdown4.6 KB
references/polars-api.mdmarkdown4.3 KB
rules/charts.ko.mdmarkdown6.2 KB
rules/charts.mdmarkdown5.3 KB
rules/engine-selection.ko.mdmarkdown7.0 KB
rules/engine-selection.mdmarkdown6.1 KB
rules/placement.ko.mdmarkdown5.5 KB
rules/placement.mdmarkdown4.5 KB
rules/query-evidence.ko.mdmarkdown5.9 KB
rules/query-evidence.mdmarkdown5.3 KB
scripts/probe-engines.shexecutablescript8.2 KB

references/polars-api.md

View on GitHub
# Polars API Notes (1.x)

Use this reference before writing a Polars pipeline. Spelled-in-full current names are listed beside the older spellings that training data teaches, because the older spellings fail, warn, or silently change meaning in 1.x. This reference is a snapshot, not a living document: the `last_verified_at` line records when the notes were last checked against the source snapshot, and a run that depends on a name here re-checks the live documentation when the installed version disagrees.

> `last_verified_at: 2026-11-09`

## 1. Current Names

| Use (1.x) | Not (older) |
|---|---|
| `pl.scan_csv(...)`, `pl.scan_parquet(...)`, then `.collect()` | eager `read_csv`, `read_parquet` on big files |
| `.group_by(...)` | `.groupby(...)` |
| `pl.len()` | `pl.count()` |
| `.collect(engine="streaming")` | `.collect(streaming=True)` |
| `.unpivot(...)` | `.melt(...)` |

## 2. Lazy First

```python
out = (pl.scan_csv("data.csv")
       .filter(pl.col("value") > 100)
       .group_by("category")
       .agg(pl.col("value").sum().alias("total"), pl.len().alias("n"))
       .sort("total", descending=True)
       .collect())
```

- `scan_csv` and `scan_parquet` build a plan; filters and projections push down to the file read, and the plan executes once at `collect()`.
- Eager `read_*` is for small files mutated interactively. On a file larger than memory it loads the whole frame and defeats the placement rule ([`rules/placement.md`](../rules/placement.md) §3).
- `pl.len()` counts rows in an aggregation; `pl.count()` is an older name that fails or warns in 1.x.

## 3. Streaming Past RAM

```python
out = (pl.scan_parquet("huge.parquet")
       .filter(pl.col("status") == "active")
       .group_by("region").agg(pl.len())
       .collect(engine="streaming"))
```

- Streaming executes lazy plans only. Keep the plan lazy end to end; an intermediate `collect()` breaks it into eager pieces and the chunks stop.
- `streaming=True` is the older spelling of the same flag; in 1.x it fails or warns where `engine="streaming"` works.

## 4. Reshapes

- `unpivot` melts wide columns into long form; `pivot` spreads them back. The older `melt` name is the rename target of the same operation.
- Wear reshapes where SQL would need nested CASE expressions; keep them inside the expression chain so the optimizer sees the whole pipeline.

## 5. DuckDB Handoff

Both engines speak Arrow, so a mixed pipeline pays no serialization cost:

```python
import duckdb
df = duckdb.sql("SELECT * FROM 'orders.csv' o JOIN 'items.csv' i USING (id)").pl()
shaped = df.with_columns((pl.col("qty") * pl.col("price")).alias("rev"))
duckdb.register("shaped", shaped)
out = duckdb.sql("SELECT category, SUM(rev) AS total FROM shaped GROUP BY 1").pl()
```

- `.pl()` requires pyarrow installed. Confirm it is present before the handoff; without it the call raises `ModuleNotFoundError`.
- Prefer `.pl()`. `.df()` is an alternative only when pandas is installed and the user, project, or available-engine constraints require it.

## 6. Version Drift

- The installed version decides which names work. When the probe ([`rules/engine-selection.md`](../rules/engine-selection.md) §1) shows an older Polars, the renames above may not apply; re-check the installed version's documentation and say in the report which spelling ran.
- Newer names may be added to this reference; older names are kept beside them only as a warning list, never as a recommended spelling.

## Sources

> Checked 2026-11-09.

| Claim | Source |
|---|---|
| The 1.x renames (`group_by`, `pl.len()`, `unpivot`, `collect(engine="streaming")`) and the lazy-scan recipe | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/references/polars-lane.md` (mechanism reference, re-authored) |
| The Arrow handoff recipe and the pyarrow requirement for `.pl()` | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/references/polars-lane.md` (mechanism reference, re-authored) |
| The streaming recipe and keeping the plan lazy end to end | `dbc-hbin/omp-ulw@74df40f mechanisms/data-analyst/references/polars-lane.md` (mechanism reference, re-authored) |

### Evidence grade

`VENDOR` - the Polars 1.x names are vendor facts re-authored from a source snapshot at a pinned commit. `last_verified_at: 2026-11-09` is the last check; the installed version's own documentation wins over this snapshot when they disagree.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
Agent Cost Reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall