Skip to content

benmonopoli/open-recruit / skills/tech-hiring/tech-hiring-data-engineering

tech-hiring-data-engineering

Verified

Data engineers build and maintain the pipelines, infrastructure, and systems that make data available for analytics, ML, and business intelligence. They sit between the raw data sources (applications, APIs, third parties) and the consumers

Data Engineering — Hiring Guide

How to Use This Guide

  • Sourcing: Use the title synonyms and Boolean strings in each section
  • Screening: Skill tiers: T1 = must-have, T2 = differentiator, T3 = rare upside
  • Interviewing: Use the assessment approach to design your interview loop
  • Benchmarking: Comp ranges are US market, growth-stage (Series B–D). Adjust for location, company stage, and current market

Universal Seniority Framework

Applies across all disciplines. Use as a baseline — teams vary on exact year ranges.

LevelTypical YoEWhat they ownMentorshipScope
Junior (L3 / IC1)0–2 yearsDefined tasks, bug fixes, small featuresNeeds daily guidanceSelf
Mid (L4 / IC2)2–5 yearsFeatures end-to-end, moderate ambiguityOccasional junior supportFeature / component
Senior (L5 / IC3)5–8 yearsProjects, technical decisions, cross-team coordinationActive mentorshipTeam / project
Staff (L6 / IC4)8–12 yearsTechnical strategy, cross-team systems, org-wide standardsMultiplies teamsOrg
Principal / Distinguished12+ yearsCompany-wide technical direction, external influenceDefines engineering cultureCompany

Promotion-readiness signal: Consistently operating at the next level before the title, not just meeting current-level expectations.


Data Engineering

What This Role Does

Data engineers build and maintain the pipelines, infrastructure, and systems that make data available for analytics, ML, and business intelligence. They sit between the raw data sources (applications, APIs, third parties) and the consumers (data scientists, analysts, dashboards). Poor data engineering is the primary reason ML projects fail.

Titles & Synonyms

LevelCommon titles
JuniorJunior Data Engineer, Data Infrastructure Engineer, ETL Developer
MidData Engineer, Analytics Engineer, Pipeline Engineer
SeniorSenior Data Engineer, Lead Data Engineer, Data Platform Engineer
Staff+Staff Data Engineer, Principal Data Engineer, Head of Data Platform
SpecialistStreaming Engineer (Kafka/Flink focus), Analytics Engineer (dbt focus)

Skill Tiers

Tier 1 — Must-have:

  • Python for data pipelines: data manipulation, scripting, automation
  • SQL: advanced queries, performance tuning, understanding of query plans
  • At least one orchestration tool: Airflow, Prefect, Dagster, or similar
  • Cloud data warehouse experience: BigQuery, Snowflake, or Redshift
  • Data modelling: star schema, normalisation, slowly changing dimensions

Tier 2 — Differentiator:

  • Batch processing: Apache Spark (PySpark) for large-scale transformations
  • Streaming: Kafka, Flink, Kinesis — real-time pipeline experience
  • dbt (data build tool) — increasingly standard for analytics engineering
  • Data quality frameworks: Great Expectations, dbt tests, custom validation
  • Data cataloguing and lineage: DataHub, OpenMetadata, Alation
  • Infrastructure as Code: Terraform for cloud resource management

Tier 3 — Bonus:

  • Apache Iceberg, Delta Lake, or Hudi (modern table formats)
  • Custom Spark/Flink jobs in Scala or Java
  • Cost optimisation at scale (query cost, storage tiering)
  • Contributions to open source data tools

Sourcing Strings

# LinkedIn X-Ray
site:linkedin.com/in/ ("data engineer" OR "analytics engineer" OR "pipeline engineer" OR "ETL developer") (Airflow OR Spark OR Kafka OR dbt OR Snowflake OR BigQuery) -recruiter -talent

# Streaming specialists
site:linkedin.com/in/ ("data engineer" OR "streaming engineer") (Kafka OR Flink OR Kinesis OR "real-time") -recruiter

# GitHub
language:python topic:data-engineering followers:>20 pushed:>2025-01-01
site:github.com "joined on" (Airflow OR dbt OR Kafka OR Spark OR "data pipeline") "data engineer"

Seniority Signals

  • Junior → Mid: Moves from "I build pipelines given a spec" to "I identify what data consumers need and design accordingly"
  • Mid → Senior: Proactively raises data quality, schema evolution, and cost concerns; designs systems that other engineers can maintain
  • Senior → Staff: Defines the data platform strategy; enables data scientists and analysts to self-serve; reduces pipeline toil company-wide

Interview & Assessment Loop

StageFormatWhat you're assessing
SQL screenComplex multi-join query, window functions, explain planFundamentals non-negotiable
Technical screenPython coding + pipeline designCode quality, data handling patterns
System designDesign a pipeline for [realistic scenario]Scale thinking, failure handling, idempotency
Take-home (optional)Build a small pipeline with dbt or AirflowPractical skills, code organisation
BehaviouralSTAR — working with data consumers, handling bad dataData engineering is a service role

Critical question: "What do you do when an upstream data source sends malformed data at 2am?" Tests failure handling philosophy, monitoring, alerting, and on-call readiness.

Red Flags

  • No understanding of idempotency in pipelines ("just re-run it")
  • Can't explain the difference between batch and streaming at a technical level
  • No experience with data testing or quality validation
  • "I just write SQL in a notebook" — no pipeline engineering experience
  • No awareness of schema evolution and how breaking changes propagate

Compensation (2025, US, Growth-Stage)

LevelBase salary range
Junior$95k – $135k
Mid$135k – $175k
Senior$160k – $220k
Staff$205k – $275k
Principal$245k – $320k+
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 319kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall