How does it monitor job failures?

The skill utilizes the system.lakeflow.job_run_timeline table to identify failed runs, extract error messages, and calculate job durations.

Is the billing data provided in real-time?

Billing data in system tables typically has a lag of up to 24 hours, so it is best used for trend analysis rather than real-time budget alerting.

What are the prerequisites for using this skill?

You need a Databricks Premium or Enterprise workspace with Unity Catalog enabled, along with access to the system.billing, system.compute, and system.access catalogs.

Does it support external monitoring tools like Prometheus?

Yes, it includes patterns for using the Databricks SDK to export cluster events and metrics to external pushgateways or monitoring stacks.

Can this skill help with Databricks cost optimization?

Yes, it provides specific SQL queries to track usage_quantity and estimated costs in USD across different clusters and SKUs over time.

Databricks Observability

Name: Databricks Observability
Author: micsapp

bymicsapp

0•

Analytics & Monitoring

Implements comprehensive monitoring, performance tracking, and alerting for Databricks jobs and infrastructure using system tables.

The Databricks Observability skill enables developers and data engineers to automate the setup of robust monitoring for Databricks environments. It leverages Unity Catalog system tables and the Databricks SDK to track job health, analyze cluster utilization and DBU costs, and identify performance bottlenecks in SQL warehouses. This skill is essential for teams looking to maintain operational excellence, optimize cloud spending, and ensure the reliability of data pipelines through automated alerting and metric exports.

Key Features

01Performance analysis for SQL warehouses and slow query detection

02Metric export patterns via Databricks SDK for external monitoring stacks

03Automated job health monitoring using system.lakeflow tables

04Detailed DBU consumption and cost tracking by cluster and SKU

050 GitHub stars

06Automated alerting configuration for pipeline failures and latency

Use Cases

01Setting up automated dashboards to monitor production job failures and durations

02Configuring real-time alerts for critical data pipeline health and resource limits

03Optimizing cloud spend by identifying underutilized clusters and high-cost SQL queries

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add micsapp/micstec-skills databricks-observability

For use in Claude.ai and ChatGPT

Key Features

01Performance analysis for SQL warehouses and slow query detection

02Metric export patterns via Databricks SDK for external monitoring stacks

03Automated job health monitoring using system.lakeflow tables

04Detailed DBU consumption and cost tracking by cluster and SKU

050 GitHub stars

06Automated alerting configuration for pipeline failures and latency

Use Cases

01Setting up automated dashboards to monitor production job failures and durations

02Configuring real-time alerts for critical data pipeline health and resource limits

03Optimizing cloud spend by identifying underutilized clusters and high-cost SQL queries

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add micsapp/micstec-skills databricks-observability

For use in Claude.ai and ChatGPT