Data Pipeline Architect
Architects scalable ETL/ELT data pipelines with robust orchestration, error handling, and automated data quality validation.
The Data Pipeline Architect skill empowers Claude to design production-grade data engineering workflows that move data reliably from source to destination. It provides expert guidance on choosing between ETL and ELT architectures, implementing batch or streaming processes, and ensuring data integrity through advanced validation frameworks and monitoring. Whether building with Airflow, dbt, or Kafka, this skill helps users implement industry best practices for idempotency, retry logic, and schema drift handling to ensure high-performance, resilient data infrastructure.
主な機能
01Designing ETL/ELT architecture patterns for cloud and local environments
02Incremental processing and Change Data Capture (CDC) implementations
03Automated data quality validation and monitoring frameworks
042 GitHub stars
05Error handling patterns including Dead Letter Queues and backoffs
06Implementing pipeline orchestration strategies for Airflow, dbt, and Prefect
ユースケース
01Migrating legacy ETL processes to modern cloud ELT using dbt and Snowflake
02Designing real-time event-driven streaming pipelines for clickstream data
03Setting up automated data reconciliation and quality checks for a data lake