How are failures managed in these patterns?

It implements detailed error handling strategies using failure callbacks, custom trigger rules, and integration patterns for monitoring tools.

How does this skill help with Airflow 2.0+ features?

It provides specific patterns for the TaskFlow API and decorators, allowing you to write cleaner Pythonic code instead of traditional operators.

Does it support cloud-specific sensors?

The skill includes patterns for AWS S3 sensors, external task sensors, and custom API sensors using 'reschedule' mode to optimize worker resources.

Can I use this for dynamic pipeline generation?

Yes, it includes factory patterns to generate multiple DAGs from configuration files or metadata, significantly reducing manual code overhead.

Airflow DAG Patterns

Name: Airflow DAG Patterns
Author: Tahir-yamin

byTahir-yamin

•

データサイエンスとML

Builds production-ready Apache Airflow pipelines using industry-standard patterns for orchestration, error handling, and automated testing.

This skill provides a comprehensive library of reusable patterns for Apache Airflow, enabling developers to construct robust data engineering workflows. It covers modern features like the TaskFlow API, dynamic DAG generation, and complex branching logic while ensuring pipelines remain idempotent, atomic, and observable. Whether you are automating ETL jobs or managing complex cross-DAG dependencies, this skill offers the implementation patterns and best practices needed to scale Airflow in production environments.

主な機能

01Advanced sensor implementations for S3, external tasks, and custom APIs

02Dynamic DAG factory patterns for managing multi-source pipelines at scale

033 GitHub stars

04Modern TaskFlow API implementations for clean, Pythonic ETL logic

05Standardized testing patterns to ensure DAG reliability and import stability

06Robust error handling with task/DAG callbacks and intelligent retry strategies

ユースケース

01Implementing cross-system orchestration using S3 sensors and external task monitoring

02Building scalable ETL/ELT pipelines with complex dependencies and conditional branching logic

03Creating dynamic workflow factories to reduce boilerplate in high-volume data environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add tahir-yamin/dev-engineering-playbook airflow-dag-patterns

For use in Claude.ai and ChatGPT

Download Skill