Can this skill help with legacy pipeline migration?

Yes, it includes a checklist and anti-pattern guide to help identify and fix common issues like monolithic DAGs and hardcoded dates during refactoring.

Why is idempotency a core principle of this skill?

Idempotency ensures that running a task multiple times produces the same result, which is critical for safely retrying failed pipeline runs without duplicating data.

Does this skill handle specific cloud provider implementations?

While it focuses on general architecture, it is designed to work in tandem with cloud-specific skills like data-aws for implementation-heavy tasks.

How does this skill improve data quality?

It provides standardized patterns for data validation checks between pipeline stages and defines schema expectations through data contracts to prevent silent failures.

What orchestration tools does this skill support?

This skill focuses on universal architectural patterns applicable to major orchestration tools including Apache Airflow, AWS Step Functions, and dbt.

Data Pipeline Architecture

Name: Data Pipeline Architecture
Author: AlexanderStephenThompson

byAlexanderStephenThompson

0•

数据科学与机器学习

Implements robust ETL/ELT architectures with advanced orchestration patterns and automated data quality checks.

This skill provides comprehensive guidance for building maintainable and scalable data pipelines, focusing on industry best practices like idempotency, atomicity, and data contracts. It assists developers in designing complex orchestration workflows using tools like Airflow, dbt, or Step Functions, ensuring data integrity through automated validation layers and efficient incremental processing strategies. Whether you are architecting a new data warehouse or optimizing existing DAGs, this skill provides the architectural patterns and anti-patterns needed to prevent data corruption and ensure reliable delivery across the entire data lifecycle.

主要功能

01Idempotent write patterns and automated retry strategies

02Incremental processing and backfill strategy implementation

030 GitHub stars

04Multi-stage data lifecycle management (Raw to Aggregated)

05ETL/ELT architecture design and orchestration logic

06Integrated data quality validation and schema contracts

使用场景

01Implementing incremental loading patterns to optimize cloud compute costs

02Designing automated data validation layers to prevent downstream pipeline failures

03Architecting scalable dbt or Airflow-based data processing workflows

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add alexanderstephenthompson/claude-hub data-pipelines

For use in Claude.ai and ChatGPT

Download Skill