Metadata Manager

About

Metadata Manager is a specialized tool for automating the lifecycle of BigQuery ETL metadata, ensuring that datasets are well-documented and correctly scheduled. It generates schema and metadata files by intelligently pulling descriptions from sources like the Glean Dictionary and DataHub, and proactively identifies missing documentation. The skill also simplifies workflow orchestration by managing Airflow DAG configurations and coordinating necessary test updates whenever underlying SQL queries are modified, facilitating a seamless transition from development to production.

Key Features

• Automated generation of schema and metadata files with intelligent description sourcing
• Base schema auto-population for consistent documentation across common fields
• Proactive detection and auto-generation of missing field descriptions
• 0 GitHub stars
• Dynamic Airflow DAG creation and discovery based on standardized scheduling patterns
• Coordinated test update workflows to maintain data integrity when queries change

Use Cases

• Initializing directory structures and metadata for new BigQuery derived datasets
• Synchronizing schema documentation and field descriptions after query modifications
• Establishing and validating Airflow scheduling configurations for data pipelines