How does it handle CI/CD for Databricks?

The skill configures Databricks Asset Bundles (DABs), which allow you to manage your infrastructure, jobs, and code as a single deployable unit via YAML.

Does this skill help with data governance?

Yes, it includes automated configurations for Unity Catalog, allowing you to manage catalogs, schemas, and fine-grained access controls centrally.

What architecture pattern does this skill follow?

It implements the Medallion Architecture, which organizes data into Bronze (raw), Silver (cleansed), and Gold (curated) layers to ensure data quality and reliability.

Can I use this for Machine Learning projects?

Absolutely. It includes directory structures for MLflow integration and feature engineering within the Gold layer of the architecture.

Databricks Reference Architecture

Name: Databricks Reference Architecture
Author: micsapp

bymicsapp

0•

클라우드 인프라

Implements production-grade Databricks Lakehouse architecture utilizing Unity Catalog, Delta Lake, and the medallion pattern.

This skill provides a comprehensive framework for designing and organizing Databricks projects according to enterprise best practices. It automates the setup of the medallion architecture (Bronze, Silver, and Gold layers), configures Unity Catalog for centralized data governance, and establishes robust project structures using Databricks Asset Bundles. Ideal for data engineers and architects, it ensures that data platforms are scalable, maintainable, and ready for CI/CD integration from day one.

주요 기능

01Unity Catalog hierarchy and permission management

02Standardized project directory and naming conventions

03Databricks Asset Bundle (DAB) configuration

040 GitHub stars

05Medallion architecture (Bronze/Silver/Gold) implementation

06Automated ETL and ML pipeline templates

사용 사례

01Bootstrapping new Databricks projects with industry-standard layouts

02Standardizing data governance and CI/CD workflows across data teams

03Migrating legacy workloads to Unity Catalog and Delta Lake

주요 기능

01Unity Catalog hierarchy and permission management

02Standardized project directory and naming conventions

03Databricks Asset Bundle (DAB) configuration

040 GitHub stars

05Medallion architecture (Bronze/Silver/Gold) implementation

06Automated ETL and ML pipeline templates

사용 사례

01Bootstrapping new Databricks projects with industry-standard layouts

02Standardizing data governance and CI/CD workflows across data teams

03Migrating legacy workloads to Unity Catalog and Delta Lake