Can it help with dynamic DAG generation?

Yes, it includes specific patterns for using factory functions and configuration objects to generate multiple DAGs dynamically, reducing code duplication for similar data sources.

Does it include automated testing for DAGs?

Yes, it provides strategies for testing DAG loading (DagBag validation) and unit testing custom operator logic to prevent breaking changes during deployment.

Does this skill support Airflow 1.x?

While many core principles apply, this skill prioritizes modern Airflow 2.x+ patterns, such as the TaskFlow API and decorators, to ensure your pipelines are future-proof and maintainable.

How does it handle external system dependencies?

The skill provides patterns for S3KeySensors, ExternalTaskSensors, and custom API sensors using 'reschedule' mode to efficiently wait for data without consuming worker slots.

Apache Airflow DAG Patterns

Name: Apache Airflow DAG Patterns
Author: GaitanS

byGaitanS

0•

Ciencia de Datos y ML

Implements production-grade Apache Airflow workflows using industry-standard patterns for orchestration, error handling, and testing.

This skill equips Claude with the expertise to design and implement robust Apache Airflow DAGs (Directed Acyclic Graphs) following data engineering best practices. It focuses on modern Airflow 2.x features like the TaskFlow API and dynamic DAG generation while ensuring workflows are idempotent, atomic, and observable. Whether you are migrating legacy pipelines, setting up complex sensors for external dependencies, or implementing sophisticated error-handling callbacks and branching logic, this skill provides the templates and architectural guidance necessary for scalable, production-ready orchestration.

Características Principales

01Standardized testing strategies for DAG integrity and logic validation

02Modern TaskFlow API implementation for clean, decorator-based Pythonic workflows

03Robust error handling using custom failure callbacks and trigger rules

04Dynamic DAG generation patterns for managing high-volume, similar data pipelines

05Advanced sensor configurations for S3, APIs, and cross-DAG dependencies

060 GitHub stars

Casos de Uso

01Creating self-healing workflows with sophisticated retries and alerting systems

02Building idempotent ETL/ELT pipelines with incremental loading and backfill support

03Developing complex orchestration logic involving branching and conditional task execution

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add gaitans/ai-mancare airflow-dag-patterns

For use in Claude.ai and ChatGPT

Download Skill