Medallion Schema Reference

About

This skill streamlines the development of PySpark transformation pipelines within medallion architecture data lakes. By proactively querying DuckDB warehouses and parsing data dictionaries, it ensures that generated code adheres to exact column names, data types, and complex business rules. It bridges the gap between raw data sources and analytical layers by identifying mapping conventions, handling deduplication strategies, and implementing standard ETL patterns, ultimately reducing schema-related errors and manual debugging during data engineering workflows.

Key Features

• Business logic extraction from markdown-based data dictionaries
• Standardized ETL class pattern generation with built-in error handling
• Cross-layer schema comparison to identify required transformations and mappings
• Automated DuckDB schema querying for exact column names and data types
• 0 GitHub stars
• Medallion architecture naming convention and mapping validation

Use Cases

• Creating analytical Gold layer tables by joining multiple Silver data sources
• Generating PySpark transformation code to move data from Bronze to Silver layers
• Updating existing ETL pipelines when source schemas or business rules change