code-yeongyu/oh-my-openagent / packages/shared-skills/skills/data-scientist
data-scientist
FlaggedProcesses and analyzes data with resident-kernel engines (DuckDB, Polars) and one-shot tools. Use for CSV/parquet/JSON analysis, group-by/join/aggregation, time series, distributions, cleaning, or plotting a dataset.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.9 KB |
| references/execution-surfaces.md | markdown | 4.0 KB |
| references/placement.md | markdown | 3.7 KB |
| references/polars-lane.md | markdown | 3.5 KB |
| references/uv-setup.md | markdown | 2.8 KB |
| references/visualization.md | markdown | 4.0 KB |
| scripts/ensure-js-deps.shexecutable | script | 983 B |
| scripts/ensure-py-deps.shexecutable | script | 1.3 KB |
| scripts/quick-query.py | script | 3.3 KB |
| scripts/setup-uv.ps1 | script | 2.3 KB |
| scripts/setup-uv.sh | script | 1.8 KB |
references/placement.md
View on GitHub# Placement: where should this computation live?
Decide before touching the data. Wrong placement wastes minutes (re-scanning a file queried
ten times) or kills the machine (loading a dataset larger than RAM and swapping).
## The probe (run first, once)
Three facts, one cell or script:
```python
import os, re, shutil, subprocess, sys
size = os.path.getsize("data.csv") # bytes on disk
disk_free = shutil.disk_usage(".").free # spill headroom
if sys.platform == "darwin": # free RAM = free + inactive + speculative pages
vm = subprocess.run(["vm_stat"], capture_output=True, text=True).stdout
page = int(re.search(r"page size of (\d+)", vm).group(1))
ram_free = page * sum(int(re.search(rf"{k}:\s+(\d+)", vm).group(1))
for k in ("Pages free", "Pages inactive", "Pages speculative"))
else: # Linux: MemAvailable
ram_free = next(int(line.split()[1]) * 1024 for line in open("/proc/meminfo")
if line.startswith("MemAvailable:"))
# row estimate without loading (DuckDB streams the scan):
# duckdb.sql("SELECT count(*) FROM 'data.csv'")
```
CSV typically expands 2-5x in memory (string columns dominate); Parquet expands less
predictably — compressed columns can inflate 10x. Estimate the working set from the
decompressed size of the columns the question actually touches, not the file size.
## In memory — load once, iterate
When the working set stays within roughly 25% of free RAM AND the session will run repeated
queries: load once (`CREATE TABLE t AS SELECT ...` in DuckDB, or a collected DataFrame),
then iterate. One scan up front converts every later query from a file re-scan into
milliseconds. Prune at load time — select only the needed columns, filter obvious dross —
so the resident table is the working set, not the raw file.
## In place / streaming — single pass, or bigger than RAM
- Single-pass questions: query the file directly (`FROM 'data.csv'`). Loading first is pure
waste.
- Bigger than RAM, SQL-shaped: cap DuckDB and let it spill —
```sql
SET memory_limit = '4GB';
SET temp_directory = '/tmp/duckdb_spill';
```
Aggregations, sorts, and window functions run out-of-core: slower, but bounded.
- Bigger than RAM, DataFrame-shaped: Polars streaming (`collect(engine="streaming")` on a
lazy plan) in the resident kernel — or a uv one-shot on kernel-less harnesses.
- Manual chunked loops (read N rows, process, repeat) are the last resort — the engines'
own out-of-core paths are faster and simpler than hand-rolled chunking.
## Remote, in place — move the query to the data
- Files behind http(s)/S3: DuckDB's httpfs extension reads Parquet and CSV remotely with
projection and predicate pushdown —
```sql
INSTALL httpfs; LOAD httpfs; -- one-time per environment
SELECT region, SUM(amount) FROM 'https://example.com/sales.parquet'
WHERE sale_date >= '2026-01-01' GROUP BY region;
```
Only matching row groups and referenced columns cross the network, not the file.
- Data on another machine you can execute on (a remote worker with more RAM, a box closer
to the data): run the query there and return the aggregate. A group-by result is
kilobytes; the source is gigabytes.
- Decision rule: result much smaller than data — move the query. Repeated local iteration
on one slice — move a pruned copy of that slice once, then work locally in memory.
## Hardware notes
- Both engines parallelize across all cores by default; leave that alone except on shared
machines (`SET threads = N` in DuckDB, `POLARS_MAX_THREADS` for Polars).
- Sustained swapping is the failure mode to avoid on memory-tight machines: when the probe
says the working set is close to free RAM, choose streaming, not hope.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |