Data Pipeline Architecture
Implements robust ETL/ELT architectures with advanced orchestration patterns and automated data quality checks.
This skill provides comprehensive guidance for building maintainable and scalable data pipelines, focusing on industry best practices like idempotency, atomicity, and data contracts. It assists developers in designing complex orchestration workflows using tools like Airflow, dbt, or Step Functions, ensuring data integrity through automated validation layers and efficient incremental processing strategies. Whether you are architecting a new data warehouse or optimizing existing DAGs, this skill provides the architectural patterns and anti-patterns needed to prevent data corruption and ensure reliable delivery across the entire data lifecycle.