Is this skill suitable for junior developers?

Absolutely; it provides senior-level architectural guidance, mentorship patterns, and documentation-as-code practices to help developers of all levels implement industry standards.

Does it focus on batch or real-time processing?

It provides expertise for both, including horizontal scaling for batch processing and high-throughput, low-latency architectures for real-time streaming data.

Can this skill help with MLOps and model deployment?

Yes, it includes production patterns for ML model deployment, feature store integration, and real-time inference systems.

How does this skill handle data governance?

The skill incorporates PII handling, anonymization, GDPR/CCPA compliance, and security auditing into the data pipeline design process.

What technologies does the Senior Data Engineer skill support?

It covers a modern data stack including Python, SQL, Spark, Airflow, dbt, Kafka, Databricks, and major cloud providers like AWS, GCP, and Azure.

Senior Data Engineer

Name: Senior Data Engineer
Author: pur3v4d3r

bypur3v4d3r

•

Data Science & ML

Builds scalable, production-grade data pipelines and ETL/ELT systems using the modern data stack.

About

This skill equips Claude with the expertise of a senior data engineer to design, implement, and optimize enterprise-scale data architectures. It provides comprehensive guidance on building fault-tolerant ETL/ELT pipelines, managing distributed computing frameworks like Spark, and implementing DataOps practices for seamless data lifecycle management. Whether you are architecting real-time streaming systems with Kafka or modeling data warehouses with dbt and Snowflake, this skill ensures best practices in scalability, security, and performance are applied to every data infrastructure task.

Key Features

1 GitHub stars
End-to-end ETL/ELT pipeline orchestration with Airflow and dbt
Real-time data streaming and high-throughput inference architectures
DataOps integration for automated testing, monitoring, and quality assurance
Distributed data processing strategies using Spark and Databricks
Advanced data modeling patterns for scalable data warehousing

Use Cases

Migrating legacy data workflows to modern distributed cloud infrastructure
Designing high-availability data architectures for production AI/ML systems
Implementing comprehensive data quality and observability frameworks

About

Key Features

1 GitHub stars
End-to-end ETL/ELT pipeline orchestration with Airflow and dbt
Real-time data streaming and high-throughput inference architectures
DataOps integration for automated testing, monitoring, and quality assurance
Distributed data processing strategies using Spark and Databricks
Advanced data modeling patterns for scalable data warehousing

Use Cases

Migrating legacy data workflows to modern distributed cloud infrastructure
Designing high-availability data architectures for production AI/ML systems
Implementing comprehensive data quality and observability frameworks