How does it handle error notification and callbacks?

It provides templates for implementing task-level and DAG-level failure callbacks to integrate with Slack, email, PagerDuty, or other monitoring tools.

Can I use this for dynamic DAG generation?

Absolutely. The skill includes patterns for creating DAG factories that generate multiple workflows from configuration files or database entries.

Does it include patterns for testing DAGs?

Yes, it covers essential testing strategies including DAG loading tests using Pytest and DagBag to ensure your code is valid before deployment.

Does this skill support Airflow 2.0+ features?

Yes, it specifically focuses on modern Airflow 2.x patterns including the TaskFlow API, decorators like @dag and @task, and updated provider interfaces.

Are the patterns compatible with cloud-native Airflow versions?

Yes, the patterns work across self-hosted Airflow, Managed Workflows for Apache Airflow (MWAA) on AWS, and Google Cloud Composer.

Airflow DAG Patterns

Name: Airflow DAG Patterns
Author: EngineerWithAI

byEngineerWithAI

0•

데이터 과학 및 ML

Builds production-ready Apache Airflow DAGs using industry-standard patterns for orchestration and data engineering.

This skill provides comprehensive guidance and boilerplate for creating scalable Apache Airflow workflows. It covers modern Airflow features like the TaskFlow API, dynamic DAG generation, sophisticated branching logic, and robust sensor implementations. Whether you are building simple ETL pipelines or complex data platforms, this skill ensures your DAGs follow best practices for idempotency, atomicity, and error handling, while also providing testing strategies to ensure pipeline reliability before deployment in production environments.

주요 기능

01Advanced orchestration logic including branching and conditional execution

02Sensor and dependency patterns for cross-DAG and external event synchronization

030 GitHub stars

04Production-grade error handling, alerting, and failure callbacks

05Dynamic DAG factory templates for scaling multiple similar pipelines

06Modern TaskFlow API implementation patterns for clean, Pythonic DAGs

사용 사례

01Orchestrating complex machine learning model training workflows

02Automating recurring data infrastructure tasks across cloud providers

03Designing idempotent and atomic ETL/ELT data pipelines

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add engineerwithai/engineerwith-agents airflow-dag-patterns

For use in Claude.ai and ChatGPT

Download Skill