How does this skill handle Databricks governance?

It provides templates for Unity Catalog to manage fine-grained access control, environment isolation, and three-level namespace governance (Catalog.Schema.Table).

Does it include performance optimization features?

Yes, it incorporates best practices for Delta Lake maintenance, including Z-Ordering, OPTIMIZE commands, and automated VACUUM schedules to keep queries fast.

What is the Medallion Architecture implemented by this skill?

It implements a three-layer data design (Bronze/Raw, Silver/Cleaned, Gold/Curated) that progressively improves data quality and structure for downstream analytics and ML.

Can I use this skill for automated deployments?

Yes, it includes configurations for Databricks Asset Bundles (DABs) which allow you to define infrastructure and code as a bundle for automated CI/CD pipelines.

Databricks Reference Architecture

Name: Databricks Reference Architecture
Author: jeremylongshore

byjeremylongshore

•

1,901

•

数据库管理

Implements production-ready Databricks Lakehouse architectures using Unity Catalog and the medallion pattern.

This skill provides a comprehensive framework for designing and deploying Databricks environments following industry best practices. It automates the creation of workspace structures, Unity Catalog governance models, and three-tier (Bronze, Silver, Gold) data processing pipelines. It is particularly useful when establishing new projects, reviewing existing architecture for performance and security, or implementing CI/CD workflows using Databricks Asset Bundles to ensure consistency across development, staging, and production environments.

主要功能

01Production-ready Medallion architecture implementation for structured data refinement

02Declarative CI/CD configuration using Databricks Asset Bundles

03Performance-optimized compute strategies for ETL jobs and SQL warehouses

04Built-in Delta Lake maintenance routines including OPTIMIZE and VACUUM schedules

05Automated Unity Catalog hierarchy setup with environment-isolated catalogs and schemas

061,901 GitHub stars

使用场景

01Modernizing legacy Spark workloads into a Unity Catalog-managed Lakehouse

02Bootstrapping a new Databricks workspace with enterprise-grade governance and security

03Establishing standardized project structures for collaborative data engineering teams

主要功能

01Production-ready Medallion architecture implementation for structured data refinement

02Declarative CI/CD configuration using Databricks Asset Bundles

03Performance-optimized compute strategies for ETL jobs and SQL warehouses

04Built-in Delta Lake maintenance routines including OPTIMIZE and VACUUM schedules

05Automated Unity Catalog hierarchy setup with environment-isolated catalogs and schemas

061,901 GitHub stars

使用场景

01Modernizing legacy Spark workloads into a Unity Catalog-managed Lakehouse

02Bootstrapping a new Databricks workspace with enterprise-grade governance and security

03Establishing standardized project structures for collaborative data engineering teams