Can this skill help with legacy pipeline migration?

Yes, it includes a checklist and anti-pattern guide to help identify and fix common issues like monolithic DAGs and hardcoded dates during refactoring.

Why is idempotency a core principle of this skill?

Idempotency ensures that running a task multiple times produces the same result, which is critical for safely retrying failed pipeline runs without duplicating data.

Does this skill handle specific cloud provider implementations?

While it focuses on general architecture, it is designed to work in tandem with cloud-specific skills like data-aws for implementation-heavy tasks.

How does this skill improve data quality?

It provides standardized patterns for data validation checks between pipeline stages and defines schema expectations through data contracts to prevent silent failures.

What orchestration tools does this skill support?

This skill focuses on universal architectural patterns applicable to major orchestration tools including Apache Airflow, AWS Step Functions, and dbt.

Data Pipeline Architecture

Name: Data Pipeline Architecture
Author: AlexanderStephenThompson

byAlexanderStephenThompson

データサイエンスとML

Implements robust ETL/ELT architectures with advanced orchestration patterns and automated data quality checks.

概要

This skill provides comprehensive guidance for building maintainable and scalable data pipelines, focusing on industry best practices like idempotency, atomicity, and data contracts. It assists developers in designing complex orchestration workflows using tools like Airflow, dbt, or Step Functions, ensuring data integrity through automated validation layers and efficient incremental processing strategies. Whether you are architecting a new data warehouse or optimizing existing DAGs, this skill provides the architectural patterns and anti-patterns needed to prevent data corruption and ensure reliable delivery across the entire data lifecycle.

主な機能

Idempotent write patterns and automated retry strategies
Incremental processing and backfill strategy implementation
0 GitHub stars
Multi-stage data lifecycle management (Raw to Aggregated)
ETL/ELT architecture design and orchestration logic
Integrated data quality validation and schema contracts

ユースケース

Implementing incremental loading patterns to optimize cloud compute costs
Designing automated data validation layers to prevent downstream pipeline failures
Architecting scalable dbt or Airflow-based data processing workflows

概要

主な機能

Idempotent write patterns and automated retry strategies
Incremental processing and backfill strategy implementation
0 GitHub stars
Multi-stage data lifecycle management (Raw to Aggregated)
ETL/ELT architecture design and orchestration logic
Integrated data quality validation and schema contracts

ユースケース

Implementing incremental loading patterns to optimize cloud compute costs
Designing automated data validation layers to prevent downstream pipeline failures
Architecting scalable dbt or Airflow-based data processing workflows