How does the skill handle data quality issues?

It helps implement a comprehensive data quality framework that performs checks at every stage, including row count completeness, timestamp freshness, and schema validation.

Can it suggest specific tools for pipeline orchestration?

Yes, it provides implementation patterns and best practices for modern orchestration tools like Airflow, Dagster, Prefect, and dbt.

How does this skill help with ETL vs ELT decisions?

It evaluates your specific compute infrastructure, data volume, and transformation complexity to recommend whether to transform data before or after loading into the target warehouse.

Does it support real-time data streaming?

Yes, the skill includes specialized design patterns for streaming architectures using technologies like Kafka, Flink, and Spark Streaming for low-latency requirements.

Data Pipeline Architect

Name: Data Pipeline Architect
Author: organvm-iv-taxis

byorganvm-iv-taxis

•

Database Management

Architects scalable ETL/ELT data pipelines with robust orchestration, error handling, and automated data quality validation.

The Data Pipeline Architect skill empowers Claude to design production-grade data engineering workflows that move data reliably from source to destination. It provides expert guidance on choosing between ETL and ELT architectures, implementing batch or streaming processes, and ensuring data integrity through advanced validation frameworks and monitoring. Whether building with Airflow, dbt, or Kafka, this skill helps users implement industry best practices for idempotency, retry logic, and schema drift handling to ensure high-performance, resilient data infrastructure.

Key Features

01Designing ETL/ELT architecture patterns for cloud and local environments

02Incremental processing and Change Data Capture (CDC) implementations

03Automated data quality validation and monitoring frameworks

042 GitHub stars

05Error handling patterns including Dead Letter Queues and backoffs

06Implementing pipeline orchestration strategies for Airflow, dbt, and Prefect

Use Cases

01Migrating legacy ETL processes to modern cloud ELT using dbt and Snowflake

02Designing real-time event-driven streaming pipelines for clickstream data

03Setting up automated data reconciliation and quality checks for a data lake

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add organvm-iv-taxis/a-i--skills data-pipeline-architect

For use in Claude.ai and ChatGPT

Download Skill