What is the difference between ETL and ELT in this skill?

The skill helps you choose ETL when transformation is needed before loading due to limited warehouse compute, and ELT for modern cloud warehouses like Snowflake or BigQuery where compute is scalable.

How does it handle data quality and errors?

It provides patterns for staging, deduplication using SQL, and error handling strategies such as Dead Letter Queues, quarantine areas, and exponential backoff retries.

Can this skill help with real-time data processing?

Yes, it includes patterns for streaming architectures using technologies like Kafka and Kinesis, focusing on event-driven design and exactly-once semantics.

What are the core design principles of this skill?

The skill emphasizes five key principles: Idempotency (safe to re-run), Incremental processing, Observability (metrics/logs), Testability, and Recoverability.

Pipeline Design

Name: Pipeline Design
Author: timequity

bytimequity

0•

数据库管理

Architects robust ETL/ELT data pipelines with standardized patterns for reliability and scalability.

The Pipeline Design skill empowers developers and data engineers to architect high-performance data integration workflows using industry-standard ETL and ELT patterns. It provides actionable guidance on batch and streaming architectures, ensuring data integrity through idempotent operations, incremental loading strategies, and structured staging patterns. Whether you are building real-time event-driven systems with Kafka or batch processing for modern data warehouses like Snowflake and BigQuery, this skill helps implement reliable error handling and observability to maintain production-grade data pipelines.

主要功能

010 GitHub stars

02Batch and streaming pipeline pattern blueprints

03Comprehensive error handling and recovery strategies

04Standardized ETL vs ELT architecture selection

05Implementation of idempotent and incremental loading

06Advanced SQL staging and deduplication patterns

使用场景

01Modernizing legacy batch jobs with incremental loading and observability

02Designing a scalable ELT process for modern cloud data warehouses

03Implementing real-time event processing with exactly-once semantics

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add timequity/plugins pipeline-design

For use in Claude.ai and ChatGPT

Download Skill