When should I apply the Data Engineering skill?

Apply this skill whenever you are designing or reviewing systems where data movement, transformation, warehousing, or data lakes are primary concerns.

Does it support streaming data systems?

Yes, it provides specific guidance for event streaming platforms like Kafka or Kinesis, including handling out-of-order data and using event-time watermarks.

How does this skill improve data quality?

It enforces a validation-first mindset, requiring specific quality rules like completeness and uniqueness checks at ingestion to prevent propagating malformed records.

What is idempotent processing in this context?

It ensures that running a pipeline multiple times with the same input produces the same output, preventing duplicate records and ensuring reliable disaster recovery.

Can it help with schema migrations?

Absolutely. It offers patterns for additive, backward-compatible schema evolution and advises on the use of schema registries for enforcement.

Data Engineering

Name: Data Engineering
Author: ragnarula

byragnarula

0•

データベース管理

Standardizes data pipeline design, ETL processes, and schema evolution using data-as-a-product principles.

The Data Engineering skill equips Claude with a rigorous architectural mindset for designing, reviewing, and implementing robust data systems. It focuses on the entire data lifecycle, prioritizing schema evolution, rigorous data quality validation, and idempotent processing. By applying this skill, developers can ensure their data warehouses, streaming topologies, and batch pipelines are resilient to upstream failures, support backfill operations, and maintain clear documentation of data lineage and consumer contracts.

主な機能

01Idempotent processing implementation to ensure deterministic pipeline outputs

02Strategies for handling late, missing, or out-of-order data in event streams

03Comprehensive documentation of data lineage and recovery procedures

040 GitHub stars

05Schema design and evolution guidance with backward/forward compatibility strategies

06Data quality validation patterns for ingestion, transformation, and quarantine logic

ユースケース

01Reviewing Kafka stream topologies for schema compatibility and event-time handling

02Implementing automated data quality monitors and validation checks for raw ingestion

03Designing a resilient ELT pipeline for a modern data warehouse

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add ragnarula/cc-plugins data-engineering

For use in Claude.ai and ChatGPT

Download Skill