davila7/claude-code-templates / cli-tool/components/skills/development/senior-data-engineer
senior-data-engineer
VerifiedWorld-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, or implementing data governance.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.4 KB |
| references/data_modeling_patterns.md | markdown | 1.4 KB |
| references/data_pipeline_architecture.md | markdown | 1.4 KB |
| references/dataops_best_practices.md | markdown | 1.4 KB |
| scripts/data_quality_validator.pyexecutable | script | 2.7 KB |
| scripts/etl_performance_optimizer.pyexecutable | script | 2.7 KB |
| scripts/pipeline_orchestrator.pyexecutable | script | 2.7 KB |
references/data_modeling_patterns.md
View on GitHub# Data Modeling Patterns
## Overview
World-class data modeling patterns for senior data engineer.
## Core Principles
### Production-First Design
Always design with production in mind:
- Scalability: Handle 10x current load
- Reliability: 99.9% uptime target
- Maintainability: Clear, documented code
- Observability: Monitor everything
### Performance by Design
Optimize from the start:
- Efficient algorithms
- Resource awareness
- Strategic caching
- Batch processing
### Security & Privacy
Build security in:
- Input validation
- Data encryption
- Access control
- Audit logging
## Advanced Patterns
### Pattern 1: Distributed Processing
Enterprise-scale data processing with fault tolerance.
### Pattern 2: Real-Time Systems
Low-latency, high-throughput systems.
### Pattern 3: ML at Scale
Production ML with monitoring and automation.
## Best Practices
### Code Quality
- Comprehensive testing
- Clear documentation
- Code reviews
- Type hints
### Performance
- Profile before optimizing
- Monitor continuously
- Cache strategically
- Batch operations
### Reliability
- Design for failure
- Implement retries
- Use circuit breakers
- Monitor health
## Tools & Technologies
Essential tools for this domain:
- Development frameworks
- Testing libraries
- Deployment platforms
- Monitoring solutions
## Further Reading
- Research papers
- Industry blogs
- Conference talks
- Open source projects
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |