What performance optimizations are included?

The skill includes templates for Delta Lake optimization techniques such as Z-Ordering, Liquid Clustering, partition strategies, and automated file compaction.

How does this skill help with data quality?

It provides implementation patterns for quality gates, including schema enforcement, null value checks, and business rule validation to ensure only clean data reaches the Silver and Gold layers.

What is the Medallion Architecture?

It is a data design pattern that organizes data into three layers: Bronze (raw ingested data), Silver (refined and cleansed data), and Gold (curated business-level aggregates).

Does it support real-time data processing?

Yes, the skill includes patterns for incremental processing using Spark Structured Streaming and Delta Lake's streaming capabilities for low-latency updates.

Can I use this for existing Databricks environments?

Absolutely. This skill provides the architectural templates and best practices to standardize existing pipelines or migrate legacy ETL logic to a modern lakehouse structure.

Medallion Architecture for Databricks

Name: Medallion Architecture for Databricks
Author: vivekgana

byvivekgana

•

数据库管理

Implements scalable data lakehouse patterns using Bronze, Silver, and Gold layer design principles and templates.

This skill provides comprehensive guidance and boilerplate for building a robust Medallion (multi-hop) architecture on Databricks. It enables developers to structure data pipelines through raw, refined, and curated layers, incorporating best practices for data quality, schema enforcement, and performance optimization. By providing standardized implementation patterns for both batch and incremental processing, it helps teams migrate from legacy ETL to modern lakehouse architectures while ensuring data integrity, auditability, and high-performance analytics.

主要功能

01Integrated data quality validation frameworks for each pipeline stage

02Support for both batch and structured streaming ingestion patterns

032 GitHub stars

04Multi-layered design patterns for Bronze, Silver, and Gold data zones

05Pre-configured templates for incremental processing and Delta Lake merges

06Optimization strategies including Z-Order, Liquid Clustering, and Auto-Optimize

使用场景

01Establishing automated data quality gates and governance across data layers

02Migrating traditional data warehouses to a modern Lakehouse architecture

03Building scalable production-grade data pipelines on Databricks

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add vivekgana/databricks-platform-marketplace medallion-architecture

For use in Claude.ai and ChatGPT

Download Skill