Does it support specific orchestration tools?

Yes, it provides implementation patterns and best practices for leading tools such as Apache Airflow, Dagster, Prefect, and dbt.

Can it help with data quality and validation?

Absolutely. It includes frameworks for completeness checks, freshness monitoring, and reconciliation between source and target systems.

What types of data pipelines can this skill design?

It supports a wide range of architectures including periodic batch pipelines, real-time streaming event-driven systems, and hybrid Lambda or Kappa architectures.

How does this skill ensure data reliability?

It focuses on building resilient systems using idempotency, checkpointing, automated retries with backoff, and dead-letter queues for poison messages.

Data Pipeline Architect

Name: Data Pipeline Architect
Author: organvm-iv-taxis

byorganvm-iv-taxis

•

데이터베이스 관리

Designs and implements robust ETL/ELT data pipelines with integrated orchestration, error handling, and data quality validation.

The Data Pipeline Architect skill empowers Claude to design scalable, production-grade data architectures that move information reliably from source to destination. It provides comprehensive guidance on selecting between ETL and ELT patterns, implementing idempotent loading strategies, and establishing multi-stage data quality frameworks. Whether you are building batch processes with Airflow or real-time streams with Kafka, this skill ensures pipeline resilience through sophisticated error-handling patterns like dead-letter queues, automated reconciliation, and schema drift detection.

주요 기능

01Implementation of idempotent and incremental loading strategies to prevent data duplication

02Architectural selection between batch, streaming, and hybrid Lambda/Kappa patterns

03Multi-stage data quality validation for completeness, freshness, and integrity

04Optimization for modern orchestration tools like Airflow, Dagster, and dbt

05Comprehensive error handling design including retries with backoff and dead-letter queues

062 GitHub stars

사용 사례

01Designing a modern cloud data warehouse ingestion pipeline using ELT and dbt transformations

02Architecting a real-time event streaming system for sub-second data processing and analytics

03Retrofitting legacy batch processes with automated monitoring, alerting, and data reconciliation

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add organvm-iv-taxis/a-i--skills data-pipeline-architect

For use in Claude.ai and ChatGPT

Download Skill