MustafaKucukcoskun/Refine-agent-kit / shared/.agent/skills/data-engineer
data-engineer
VerifiedData engineering patterns for pandas, polars, DuckDB, ETL pipelines, and data quality controls.
Data Engineer Skill
1. Pandas Best Practices
- Method Chaining: Perform data transformations using method chaining (
.assign(),.pipe(),.query()) instead of creating temporary variables. Improves readability and optimizes RAM usage. - Dtype Optimization (Pandas 2.0+): Use
dtype_backend='numpy_nullable'to leverage PyArrow-backed nullable types likeInt64,Float64,boolean,string. Prevents errors caused by missing data (NaNvsNone). Use.convert_dtypes()to automatically switch to the best nullable dtype. - Chunking / Memory Mapping: For large files (that strain RAM), use
pd.read_csv("file.json", lines=True, chunksize=1000)ormemory_map=True(pd.read_table). - Grouping & Merging: Prefer
as_index=Falseingroupbyoperations to prevent unnecessary indexing.
2. Polars Migration and Comparison
- Lazy Evaluation: While Pandas is "eager" (execute immediately), Polars is "lazy" by default. For data pipelines requiring higher performance (>100MB), build the graph with
pl.scan_csv()and call.collect()at the end to save RAM and CPU. - Query Engine: Since Polars has a multi-threaded Rust core, the
pl.Expr(expression) API is orders of magnitude faster than pandas'applyloops. Use native expressions wherever possible (e.g.,.with_columns(pl.col("A") * 2)).
3. DuckDB Integration
- In-memory Analytics: Query pandas DataFrames directly with SQL using zero-copy semantics:
duckdb.query("SELECT * FROM df").df(). - Direct Parquet/CSV: Instead of loading into memory with Pandas, use disk as a database:
duckdb.sql("SELECT dept, AVG(salary) FROM read_parquet('data.parquet') GROUP BY dept").df(). This enables SQL capabilities on files that exceed RAM limits.
4. ETL (Extract, Transform, Load) Pipeline Pattern
- Each step should be an isolated Python function. Never write massive spaghetti blocks.
extract_data(path: Path) -> pd.DataFramevalidate_schema(df: pd.DataFrame) -> None(raises exception on unexpected dtype/null)transform_metrics(df: pd.DataFrame) -> pd.DataFrame(used as.pipe(transform_metrics))load_data(df: pd.DataFrame, target: str) -> None
- Clean up unnecessary variables between steps with
del var_nameandgc.collect()to prevent memory bloat.
5. Data Quality
- Missing Values: Filling with
fillna()using mean/median/constant is not always correct. If the feature targets machine learning, consider creatingis_nullindicator columns that note the status. For analytics, evaluatedropna()(hard delete) tradeoffs. - Outlier Detection: Use the 3 Sigma rule or IQR and perform EDA (Exploratory Data Analysis) to determine whether extreme values are actual data corruption (sensor reading errors) or rare natural events.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 318k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 117k | Updated | Claude CodeCodex | Verified | Install |