Query Writer

About

Streamlines the creation and maintenance of data transformation logic within the Mozilla BigQuery ETL ecosystem by automating the generation of query files and ensuring adherence to strict SQL formatting and naming conventions. It enforces the mandatory synchronization of unit tests and schemas whenever queries are modified, providing a cohesive workflow for developing reliable, high-performance data pipelines. By integrating with local metadata and external sources like DataHub and the Glean Dictionary, it assists developers in constructing accurate queries with proper partitioning, incremental refresh logic, and validated business logic.

Key Features

• SQL and Python ETL generation following Mozilla-specific conventions and partitioning requirements
• Automated SQL formatting and validation using built-in bqetl tooling
• Integrated schema and metadata lookup using local files, Glean Dictionary, and DataHub
• 0 GitHub stars
• Mandatory automated test management and fixture synchronization for all query modifications
• Support for complex data patterns including incremental refreshes, CTEs, and user-level aggregations

Use Cases

• Creating new incremental data transformation pipelines for BigQuery tables
• Migrating complex data processing tasks from standard SQL to Python-based ETL scripts
• Updating existing SQL query logic while ensuring downstream schemas and tests remain in sync