Traces the origins of data across Airflow pipelines to map dependencies and identify source systems.
The Upstream Data Lineage Tracer provides specialized capabilities for data engineers to navigate and document complex data ecosystems. By analyzing Airflow DAG definitions, SQL transformations, and external connections, it identifies exactly how data flows from its original source to the final destination. This skill is invaluable for performing root-cause analysis during data quality incidents, conducting impact assessments for upstream changes, and documenting data provenance for architectural reviews.
Características Principales
01Identifies producing Airflow DAGs and specific tasks
02Generates structured lineage reports and visual flow diagrams
0377 GitHub stars
04Maps table and column-level dependencies recursively
05Traces SQL FROM clauses and external API/storage sources
06Integrates with freshness checks to assess source health
Casos de Uso
01Understanding data provenance for compliance and auditing
02Performing impact analysis before modifying upstream database schemas
03Troubleshooting data discrepancies by tracing back to the raw source
