What is the Medallion Architecture implemented by this skill?

It is a data design pattern consisting of three layers—Bronze (raw), Silver (cleansed), and Gold (aggregated)—that incrementally improve data quality as it moves through the pipeline.

Can I use this skill for real-time streaming applications?

Yes, the skill provides code patterns for Spark Structured Streaming and supports incremental processing using Delta Lake's Change Data Feed (CDF).

How does the skill handle data ingestion from cloud storage?

It utilizes Databricks Auto Loader (cloudFiles) for efficient, scalable, and incremental ingestion from raw cloud storage into the Bronze layer.

Does this skill support Delta Live Tables (DLT)?

Yes, it includes specific implementation patterns and templates for building and orchestrating Delta Live Tables pipelines.

Databricks Delta Lake ETL Workflow

Name: Databricks Delta Lake ETL Workflow
Author: jeremylongshore

byjeremylongshore

•

1,243

Data Science & ML

Automates the creation of production-grade Delta Lake ETL pipelines using the Medallion Architecture pattern.

About

This skill provides a standardized framework for building Databricks data pipelines, guiding users through the implementation of the Medallion Architecture (Bronze, Silver, and Gold layers). It includes optimized patterns for raw data ingestion via Auto Loader, incremental data cleansing with Spark structured streaming, and sophisticated business aggregations for analytics-ready datasets. It is designed to help data engineers implement best practices for Unity Catalog-enabled environments and Delta Live Tables (DLT), ensuring data quality, idempotency, and scalability.

Key Features

Ready-to-use Delta Live Tables (DLT) pipeline templates
Implementation of Medallion Architecture (Bronze, Silver, Gold layers)
Automated data ingestion patterns using Databricks Auto Loader
Scalable data cleansing and PII removal logic for Silver layer
Incremental processing and upsert patterns using Delta Lake Merge
1,243 GitHub stars

Use Cases

Designing scalable data transformation pipelines with Spark and Delta Lake
Building automated data ingestion from cloud landing zones to Bronze tables
Creating aggregated, business-ready datasets for BI and analytics platforms

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-core-workflow-a

For use in Claude.ai and ChatGPT

Download Skill

GitHub