How does this skill handle data quality?

The skill follows a strict checklist including input validation for schemas and types, duplicate detection, and automated error record capturing to ensure high data integrity.

Can I use this for real-time data processing?

Yes, the skill includes specialized logic for managing Kafka consumers and producers to facilitate real-time event streaming and integration.

Does it provide status updates for long-running tasks?

Yes, the skill includes a standardized output format for status updates, tracking data quality rules, error handling progress, and test coverage.

What tech stack does the Data Engineer skill support?

It is optimized for SQL (MySQL and various Data Warehouses), Kafka for event streaming, Python or Java for pipeline code, and scheduling tools like Temporal or cron.

Data Engineer

Name: Data Engineer
Author: violetio

byvioletio

Database Management

Architects and maintains robust data pipelines, analytics infrastructure, and integration workflows.

About

The Data Engineer skill empowers Claude to act as a technical authority on data infrastructure, specializing in ETL processes, data warehouse design, and real-time event streaming with Kafka. It ensures data quality through automated validation checks and idempotent operations, enabling teams to build scalable analytics systems with comprehensive audit logging and performance optimization. This skill is ideal for projects requiring complex data transformations, database migrations, or the implementation of reliable reporting pipelines within Claude Code.

Key Features

Real-time event streaming integration using Kafka
Strict data quality validation and error handling
0 GitHub stars
Automated ETL pipeline design and implementation
Data warehouse architecture and performance tuning
Comprehensive data lineage and documentation generation

Use Cases

Implementing real-time data streaming and processing workflows
Migrating legacy data sources into modern cloud warehouses
Building idempotent data pipelines for reliable analytics

About

Key Features

Real-time event streaming integration using Kafka
Strict data quality validation and error handling
0 GitHub stars
Automated ETL pipeline design and implementation
Data warehouse architecture and performance tuning
Comprehensive data lineage and documentation generation

Use Cases

Implementing real-time data streaming and processing workflows
Migrating legacy data sources into modern cloud warehouses
Building idempotent data pipelines for reliable analytics