davila7/claude-code-templates / cli-tool/components/skills/development/senior-data-engineer
senior-data-engineer
VerifiedWorld-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, or implementing data governance.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.4 KB |
| references/data_modeling_patterns.md | markdown | 1.4 KB |
| references/data_pipeline_architecture.md | markdown | 1.4 KB |
| references/dataops_best_practices.md | markdown | 1.4 KB |
| scripts/data_quality_validator.pyexecutable | script | 2.7 KB |
| scripts/etl_performance_optimizer.pyexecutable | script | 2.7 KB |
| scripts/pipeline_orchestrator.pyexecutable | script | 2.7 KB |
scripts/data_quality_validator.py
View on GitHub#!/usr/bin/env python3
"""
Data Quality Validator
Production-grade tool for senior data engineer
"""
import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class DataQualityValidator:
"""Production-grade data quality validator"""
def __init__(self, config: Dict):
self.config = config
self.results = {
'status': 'initialized',
'start_time': datetime.now().isoformat(),
'processed_items': 0
}
logger.info(f"Initialized {self.__class__.__name__}")
def validate_config(self) -> bool:
"""Validate configuration"""
logger.info("Validating configuration...")
# Add validation logic
logger.info("Configuration validated")
return True
def process(self) -> Dict:
"""Main processing logic"""
logger.info("Starting processing...")
try:
self.validate_config()
# Main processing
result = self._execute()
self.results['status'] = 'completed'
self.results['end_time'] = datetime.now().isoformat()
logger.info("Processing completed successfully")
return self.results
except Exception as e:
self.results['status'] = 'failed'
self.results['error'] = str(e)
logger.error(f"Processing failed: {e}")
raise
def _execute(self) -> Dict:
"""Execute main logic"""
# Implementation here
return {'success': True}
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description="Data Quality Validator"
)
parser.add_argument('--input', '-i', required=True, help='Input path')
parser.add_argument('--output', '-o', required=True, help='Output path')
parser.add_argument('--config', '-c', help='Configuration file')
parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
try:
config = {
'input': args.input,
'output': args.output
}
processor = DataQualityValidator(config)
results = processor.process()
print(json.dumps(results, indent=2))
sys.exit(0)
except Exception as e:
logger.error(f"Fatal error: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |