Skip to content

code-yeongyu/oh-my-openagent / packages/shared-skills/skills/programming

programming

Unverified

Applies strict, modern language practice (typed errors, exhaustive match, tests that can fail) for Python, Rust, TypeScript, JavaScript, and Go. Use for work on .py, .rs, .ts, .js, or .go files, and whenever a test or spec file is written, reshaped, reviewed, or deleted: the four-question test authoring gate and the junk-test classes live here.

FileKindSize
SKILL.mdskill md33.3 KB
references/code-smells.mdmarkdown13.9 KB
references/go/README.mdmarkdown7.7 KB
references/go/backend-stack.mdmarkdown16.6 KB
references/go/bootstrap.mdmarkdown8.9 KB
references/go/bubbletea-v2.mdmarkdown11.3 KB
references/go/cobra-stack.mdmarkdown11.5 KB
references/go/concurrency.mdmarkdown10.9 KB
references/go/data-modeling.mdmarkdown12.0 KB
references/go/error-handling.mdmarkdown10.1 KB
references/go/golangci-strict.mdmarkdown10.0 KB
references/go/grpc-connect.mdmarkdown10.3 KB
references/go/libraries.mdmarkdown13.7 KB
references/go/one-liners.mdmarkdown5.5 KB
references/go/sqlc-pgx.mdmarkdown12.7 KB
references/go/testing.mdmarkdown14.2 KB
references/go/type-patterns.mdmarkdown9.5 KB
references/logging.mdmarkdown10.5 KB
references/python/README.mdmarkdown15.7 KB
references/python/async-anyio.mdmarkdown14.2 KB
references/python/data-modeling.mdmarkdown6.6 KB
references/python/data-processing.mdmarkdown4.2 KB
references/python/error-handling.mdmarkdown6.0 KB
references/python/fastapi-stack.mdmarkdown9.2 KB
references/python/httpx2-optimization.mdmarkdown12.0 KB
references/python/libraries.mdmarkdown10.6 KB
references/python/one-liners.mdmarkdown7.9 KB
references/python/orjson-stack.mdmarkdown13.3 KB
references/python/pydantic-ai.mdmarkdown8.2 KB
references/python/pyproject-strict.mdmarkdown9.0 KB
references/python/textual-tui.mdmarkdown5.1 KB
references/python/type-patterns.mdmarkdown4.1 KB
references/rust-ub/README.mdmarkdown9.9 KB
references/rust-ub/miri-sanitizers-loom.mdmarkdown11.5 KB
references/rust-ub/ub-taxonomy.mdmarkdown13.3 KB
references/rust/README.mdmarkdown16.6 KB
references/rust/api-design.mdmarkdown6.5 KB
references/rust/async-tokio.mdmarkdown12.6 KB
references/rust/axum-stack.mdmarkdown14.2 KB
references/rust/cargo-strict.mdmarkdown10.0 KB
references/rust/clap-stack.mdmarkdown12.1 KB
references/rust/concurrency.mdmarkdown14.3 KB
references/rust/libraries.mdmarkdown17.4 KB
references/rust/macros.mdmarkdown3.3 KB
references/rust/one-liners.mdmarkdown7.5 KB
references/rust/proptest-insta.mdmarkdown13.9 KB
references/rust/type-state.mdmarkdown13.1 KB
references/rust/unsafe-discipline.mdmarkdown12.9 KB
references/rust/zero-cost-safety.mdmarkdown16.9 KB
references/testing/README.mdmarkdown10.3 KB
references/testing/junk-classes.mdmarkdown6.2 KB
references/typescript/README.mdmarkdown10.6 KB
references/typescript/backend-hono.mdmarkdown21.2 KB
references/typescript/bootstrap.mdmarkdown6.9 KB
references/typescript/data-modeling.mdmarkdown5.0 KB
references/typescript/error-handling.mdmarkdown4.7 KB
references/typescript/oxc-lint.mdmarkdown7.6 KB
references/typescript/tsconfig-strict.mdmarkdown3.9 KB
references/typescript/type-patterns.mdmarkdown5.2 KB
scripts/go/check-no-excuse-rules.shexecutablescript7.1 KB
scripts/go/new-project.tsscript6.8 KB
scripts/go/templates/.editorconfigother206 B
scripts/go/templates/.golangci.ymlconfig1.5 KB
scripts/go/templates/AGENTS.md.tmplother845 B
scripts/go/templates/README.md.tmplother207 B
scripts/go/templates/Taskfile.ymlconfig706 B
scripts/go/templates/ci.ymlconfig863 B
scripts/go/templates/config.goscript643 B
scripts/go/templates/gitignoreother99 B
scripts/go/templates/main.go.tmplother329 B
scripts/go/templates/run.goscript350 B
scripts/python/check-no-excuse-rules.pyscript24.9 KB
scripts/python/new-project.tsscript6.7 KB
scripts/python/new-script.tsscript4.2 KB
scripts/rust/check-no-excuse-rules.shexecutablescript7.5 KB
scripts/rust/check-no-excuse-rules.test.tsscript3.4 KB
scripts/rust/new-project.tsscript6.1 KB
scripts/scaffolders.test.tsscript4.9 KB
scripts/shared/run-tool.tsscript855 B
scripts/typescript/check-no-excuse-rules.test.tsscript4.0 KB
scripts/typescript/check-no-excuse-rules.tsscript13.1 KB
scripts/typescript/check-test-junk-rules.tsscript5.5 KB
scripts/typescript/check-test-junk.test.tsscript4.9 KB
scripts/typescript/check-test-junk.tsscript5.4 KB
scripts/typescript/mutation-proof.test.tsscript1.9 KB
scripts/typescript/mutation-proof.tsscript6.4 KB
scripts/typescript/new-project.test.tsscript6.8 KB
scripts/typescript/new-project.tsscript3.1 KB
scripts/typescript/oxc/.oxfmtrc.jsonconfig256 B
scripts/typescript/oxc/.oxlintrc.jsonconfig1.7 KB
scripts/typescript/oxc/lint/oxlint-plugin/LICENSEother1.0 KB
scripts/typescript/oxc/lint/oxlint-plugin/index.jsscript288 B
scripts/typescript/oxc/lint/oxlint-plugin/no-em-dash-copy.jsscript785 B
scripts/typescript/oxc/lint/oxlint-plugin/require-suppression-reason.jsscript2.3 KB
scripts/typescript/oxc/tsconfig.jsonconfig794 B

references/python/data-processing.md

View on GitHub
# Data Processing — Polars + DuckDB

## The rule

NEVER pandas. Polars (with numpy) plus DuckDB. Pandas is 10-50x slower, has weaker types, and the modern Python data ecosystem has moved on.

## Quick decision tree

| Operation | Use | Why |
|---|---|---|
| `.csv` / `.parquet` / `.json` direct query | DuckDB | Zero memory load, SQL ergonomics |
| `.duckdb` file | DuckDB | Native format |
| Filter (any size) | Polars | row-wise filtering, low setup cost |
| Sort | Polars | in-memory sorting |
| Multi-table join | DuckDB | more join types, out-of-core joins |
| Heavy GROUP BY aggregation | DuckDB | large grouped aggregation |
| Window function | Polars | window expressions |
| Pivot / melt / string ops | Polars | reshape and string expressions |
| Larger than RAM | Polars streaming or DuckDB out-of-core | Both handle OOM |
| Mixed pipeline | Hybrid (zero-copy via Arrow) | Use each tool's strengths |

For the deep version (engine choice, placement, OOM strategies, execution templates), load the **`data-scientist`** skill - it lives in this same skill set; when the engine choice matters, measure on the actual data, as that skill says.

## Standard imports

```python
import numpy as np
import polars as pl
import duckdb
```

## DuckDB direct file query (zero memory load)

```python
result = duckdb.sql("""
    SELECT category, SUM(amount) AS total
    FROM 'data.csv'
    WHERE date >= '2026-01-01'
    GROUP BY category
    ORDER BY total DESC
""").pl()  # zero-copy → Polars DataFrame
```

`.pl()` returns Polars; `.df()` would return pandas - never use `.df()`.

## Polars lazy pipeline

```python
result = (
    pl.scan_csv("data.csv")              # lazy, no read yet
    .filter(pl.col("amount") > 1000)
    .filter(pl.col("status") == "active")
    .sort("amount", descending=True)
    .head(100)
    .collect()                            # execute optimised plan
)
```

`scan_*` over `read_*` for files; `lazy()` then `collect()` for in-memory frames. Polars optimises the entire plan before execution (predicate pushdown, projection pushdown, common subexpression elimination).

## Streaming for OOM data

```python
result = (
    pl.scan_csv("huge.csv")
    .filter(pl.col("active"))
    .group_by("category")
    .agg([
        pl.len().alias("count"),
        pl.sum("amount").alias("total"),
    ])
    .collect(engine="streaming")
)
```

## Hybrid pipeline (most realistic shape)

```python
# Phase 1: DuckDB for the join (3x faster)
joined = duckdb.sql("""
    SELECT o.*, c.region, p.category
    FROM 'orders.parquet' o
    JOIN 'customers.parquet' c ON o.customer_id = c.id
    JOIN 'products.parquet'  p ON o.product_id  = p.id
""").pl()

# Phase 2: Polars for filtering and transformation
processed = (
    joined
    .filter(pl.col("amount") > 100)
    .with_columns([
        (pl.col("amount") * 1.1).alias("amount_with_tax"),
    ])
)

# Phase 3: DuckDB for final aggregation (4x faster) - register Polars frame by name
duckdb.register("processed", processed)
final = duckdb.sql("""
    SELECT region, category, SUM(amount_with_tax) AS revenue
    FROM processed
    GROUP BY region, category
    ORDER BY revenue DESC
""").pl()
```

## Type safety with Polars

Polars supports schema overrides at read time, and `.cast()` for explicit conversion. Avoid implicit coercion in hot paths.

```python
schema = {"id": pl.Int64, "amount": pl.Float64, "date": pl.Date}
df = pl.read_csv("data.csv", schema_overrides=schema)
```

basedpyright understands `polars-stubs`, which ship with polars itself. No extra type stubs to install.

## Things you might miss from pandas (and how to do them in Polars)

| pandas | polars |
|---|---|
| `df.iloc[5]` | `df.row(5)` (named tuple) or `df[5]` (single-row frame) |
| `df.loc[df["x"] > 5]` | `df.filter(pl.col("x") > 5)` |
| `df["x"].apply(fn)` | `df["x"].map_elements(fn)` (slow path) or use native expressions |
| `df.merge(...)` | `df.join(other, on="key")` |
| `df.groupby(...).agg(...)` | `df.group_by(...).agg(...)` |
| `pd.read_csv(...).dtypes` | `pl.read_csv(...).schema` |
| `df.to_dict("records")` | `df.to_dicts()` |

## Sources

- Polars docs: <https://docs.pola.rs>
- DuckDB Python API: <https://duckdb.org/docs/api/python/overview>
- Cross-reference - this skill set's `data-scientist` skill (load it for the deep version)
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 318kUpdated Claude CodeCodexUnverifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 318kUpdated Claude CodeCodexUnverifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 318kUpdated Claude CodeCodexUnverifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 196kUpdated Claude CodeCodexVerifiedInstall