How does it handle data quality?

It integrates data quality tests and validation patterns using dbt testing frameworks and tools like Great Expectations to ensure reliable analytical outputs.

Does this skill support dbt incremental models?

Yes, it provides patterns for implementing incremental data loads and merge strategies to handle large datasets efficiently while minimizing compute costs.

Which orchestration tools are covered?

The skill includes implementation patterns for industry-standard tools like Apache Airflow, Dagster, and Prefect, helping you choose the best fit for your architecture.

Can I use this for performance optimization?

Absolutely. It includes specific migration guides for moving from pandas to polars or PySpark to handle larger data volumes and significantly increase processing speed.

Data Transformation & Pipelines

Name: Data Transformation & Pipelines
Author: ancoleman

byancoleman

•

158

数据库管理

Transforms raw data into clean, analytical assets using modern ETL/ELT patterns, SQL (dbt), and high-performance Python frameworks.

关于

This skill enables Claude to architect and implement robust data transformation workflows across the modern data stack. It provides comprehensive guidance for choosing between ETL and ELT patterns, building modular SQL models with dbt, and performing high-performance data processing with Python libraries like pandas, polars, and PySpark. Whether you are migrating legacy pipelines, implementing incremental data loads, or orchestrating complex multi-step workflows with Airflow, this skill ensures your data pipelines are production-ready, testable, and optimized for scale.

主要功能

Orchestrates complex workflows with Airflow, Dagster, or Prefect
Develops modular dbt models with staging, intermediate, and marts layers
Implements ETL and ELT transformation patterns for modern cloud warehouses
Ensures data integrity with integrated testing, validation, and quality checks
Optimizes Python data processing using polars and PySpark for high performance
158 GitHub stars

使用场景

Building incremental dbt models for large-scale cloud data warehouses
Migrating performance-bottlenecked pandas scripts to polars or PySpark
Designing production-ready Airflow DAGs with retries and dependency logic

关于

主要功能

Orchestrates complex workflows with Airflow, Dagster, or Prefect
Develops modular dbt models with staging, intermediate, and marts layers
Implements ETL and ELT transformation patterns for modern cloud warehouses
Ensures data integrity with integrated testing, validation, and quality checks
Optimizes Python data processing using polars and PySpark for high performance
158 GitHub stars

使用场景

Building incremental dbt models for large-scale cloud data warehouses
Migrating performance-bottlenecked pandas scripts to polars or PySpark
Designing production-ready Airflow DAGs with retries and dependency logic