What version of Airflow does this skill support?

This skill focuses on modern Airflow 2.0+ best practices, specifically emphasizing the TaskFlow API, decorators, and updated provider packages.

Does it include testing strategies?

The skill provides patterns for DAG integrity testing and loading validation using pytest to ensure your code is production-ready before deployment.

How does it handle complex task dependencies?

It provides implementation patterns for fan-in/fan-out structures, BranchPythonOperators, and trigger rules like NONE_FAILED_MIN_ONE_SUCCESS for complex logic.

What is the recommended way to handle external dependencies?

It demonstrates using Sensors (like S3KeySensor) and ExternalTaskSensors with 'reschedule' mode to efficiently wait for external data or upstream tasks.

Can I use this for dynamic pipeline creation?

Yes, the skill includes a dedicated factory pattern for generating multiple similar DAGs programmatically from configuration files or metadata.

Apache Airflow DAG Patterns

Name: Apache Airflow DAG Patterns
Author: pur3v4d3r

bypur3v4d3r

•

배포 및 DevOps

Builds production-ready Apache Airflow pipelines using industry best practices for orchestration, reliability, and observability.

소개

This skill empowers developers to architect robust data orchestration workflows using Apache Airflow. It provides a comprehensive library of implementation patterns, ranging from modern TaskFlow API usage and dynamic DAG generation to advanced branching logic and sophisticated error handling. Designed to ensure data pipelines are idempotent, atomic, and observable, it guides users through the complexities of sensor implementation, external dependencies, and automated testing to maintain high-quality data engineering standards in any production environment.

주요 기능

Dynamic DAG generation strategies for scaling repetitive pipeline structures
Standardized DAG design principles including idempotency and atomicity
1 GitHub stars
Advanced orchestration patterns including branching, sensors, and external triggers
Modern TaskFlow API implementation for clean, Pythonic data passing
Robust error handling frameworks with custom failure callbacks and trigger rules

사용 사례

Implementing cross-DAG dependencies and event-driven data processing
Architecting scalable ETL/ELT pipelines with complex task dependencies
Automating the deployment and validation of production-grade workflow schedules

소개

주요 기능

Dynamic DAG generation strategies for scaling repetitive pipeline structures
Standardized DAG design principles including idempotency and atomicity
1 GitHub stars
Advanced orchestration patterns including branching, sensors, and external triggers
Modern TaskFlow API implementation for clean, Pythonic data passing
Robust error handling frameworks with custom failure callbacks and trigger rules

사용 사례

Implementing cross-DAG dependencies and event-driven data processing
Architecting scalable ETL/ELT pipelines with complex task dependencies
Automating the deployment and validation of production-grade workflow schedules