Builds and optimizes scalable data pipelines, ETL/ELT systems, and robust data infrastructure.
This skill equips Claude with the expertise of a senior data engineer to architect production-grade data systems and reliable data pipelines. It provides comprehensive guidance on building batch and streaming architectures, implementing advanced data modeling patterns like Star Schemas and Data Vaults, and establishing rigorous DataOps practices. Whether you are deciding between Lambda and Kappa architectures or optimizing Spark jobs and Airflow DAGs, this skill offers the frameworks and best practices needed to ensure data quality, scalability, and performance across the modern data stack.
Key Features
01Automated data quality validation and monitoring frameworks
02Advanced data modeling for Star Schema, Snowflake, and SCD implementation
03End-to-end pipeline orchestration using Airflow, dbt, and Spark
04Architectural decision frameworks for Batch, Streaming, and Lakehouse designs
05Performance tuning for SQL queries and large-scale data processing jobs
060 GitHub stars
Use Cases
01Designing a high-throughput ETL process from a transactional database to a data warehouse
02Implementing a real-time event streaming architecture using Kafka and Flink
03Establishing a DataOps workflow with automated testing and schema validation
