When should I apply the Data Engineering skill?

Apply this skill whenever you are designing or reviewing systems where data movement, transformation, warehousing, or data lakes are primary concerns.

Does it support streaming data systems?

Yes, it provides specific guidance for event streaming platforms like Kafka or Kinesis, including handling out-of-order data and using event-time watermarks.

How does this skill improve data quality?

It enforces a validation-first mindset, requiring specific quality rules like completeness and uniqueness checks at ingestion to prevent propagating malformed records.

What is idempotent processing in this context?

It ensures that running a pipeline multiple times with the same input produces the same output, preventing duplicate records and ensuring reliable disaster recovery.

Can it help with schema migrations?

Absolutely. It offers patterns for additive, backward-compatible schema evolution and advises on the use of schema registries for enforcement.

Data Engineering

Name: Data Engineering
Author: ragnarula

byragnarula

数据库管理

Standardizes data pipeline design, ETL processes, and schema evolution using data-as-a-product principles.

关于

The Data Engineering skill equips Claude with a rigorous architectural mindset for designing, reviewing, and implementing robust data systems. It focuses on the entire data lifecycle, prioritizing schema evolution, rigorous data quality validation, and idempotent processing. By applying this skill, developers can ensure their data warehouses, streaming topologies, and batch pipelines are resilient to upstream failures, support backfill operations, and maintain clear documentation of data lineage and consumer contracts.

主要功能

Idempotent processing implementation to ensure deterministic pipeline outputs
Strategies for handling late, missing, or out-of-order data in event streams
Comprehensive documentation of data lineage and recovery procedures
0 GitHub stars
Schema design and evolution guidance with backward/forward compatibility strategies
Data quality validation patterns for ingestion, transformation, and quarantine logic

使用场景

Reviewing Kafka stream topologies for schema compatibility and event-time handling
Implementing automated data quality monitors and validation checks for raw ingestion
Designing a resilient ELT pipeline for a modern data warehouse

关于

主要功能

Idempotent processing implementation to ensure deterministic pipeline outputs
Strategies for handling late, missing, or out-of-order data in event streams
Comprehensive documentation of data lineage and recovery procedures
0 GitHub stars
Schema design and evolution guidance with backward/forward compatibility strategies
Data quality validation patterns for ingestion, transformation, and quarantine logic

使用场景

Reviewing Kafka stream topologies for schema compatibility and event-time handling
Implementing automated data quality monitors and validation checks for raw ingestion
Designing a resilient ELT pipeline for a modern data warehouse