Can I export these metrics to external tools like Grafana?

Yes, the skill includes Python patterns using the Databricks SDK to export metrics for consumption by external observability platforms like Prometheus and Grafana.

Does this skill provide real-time monitoring?

The skill uses Databricks system tables which update throughout the day; while excellent for trends, alerts, and daily health checks, there is a slight lag compared to real-time telemetry.

Can it identify slow SQL queries?

Yes, it includes performance monitoring for SQL warehouses that identifies queries exceeding duration thresholds and analyzes warehouse queue times.

How does it help with Databricks cost management?

It provides specific SQL queries that join billing data with job and cluster metadata, allowing you to calculate the exact USD cost per job run or team.

What are the prerequisites for using the Databricks Observability skill?

You need a Databricks Premium or Enterprise workspace with Unity Catalog enabled and appropriate permissions to access the 'system' catalog tables (billing, lakeflow, query, and access).

Databricks Observability & Monitoring

Name: Databricks Observability & Monitoring
Author: jeremylongshore

byjeremylongshore

•

1,844

•

Analytics & Monitoring

Configures comprehensive monitoring, metrics, and alerting for Databricks workflows and infrastructure using system tables.

This skill empowers developers and data engineers to implement full-stack observability across Databricks environments by leveraging Unity Catalog system tables. It enables automated tracking of job success rates, detailed cost-per-job analysis, SQL warehouse performance tuning, and proactive alerting via SQL or Python. By integrating cluster utilization metrics and audit log monitoring, it provides a centralized way to ensure pipeline health, optimize compute spending, and maintain security compliance within the Databricks ecosystem.

Key Features

01Automated SQL alerting configurations for Slack and email notifications

02SQL warehouse performance profiling for identifying slow queries and queue bottlenecks

03Granular cost analysis and DBU consumption tracking by cluster and SKU

04Job health monitoring for success rates and pipeline duration tracking

05Metric export patterns for integration with Prometheus, Grafana, or Datadog

061,844 GitHub stars

Use Cases

01Identifying and optimizing high-cost clusters or inefficient SQL queries to reduce cloud spend

02Building automated dashboards to monitor production data pipeline health and reliability

03Establishing a security audit trail to track user access and actions across the workspace

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-observability

For use in Claude.ai and ChatGPT

Key Features

01Automated SQL alerting configurations for Slack and email notifications

02SQL warehouse performance profiling for identifying slow queries and queue bottlenecks

03Granular cost analysis and DBU consumption tracking by cluster and SKU

04Job health monitoring for success rates and pipeline duration tracking

05Metric export patterns for integration with Prometheus, Grafana, or Datadog

061,844 GitHub stars

Use Cases

01Identifying and optimizing high-cost clusters or inefficient SQL queries to reduce cloud spend

02Building automated dashboards to monitor production data pipeline health and reliability

03Establishing a security audit trail to track user access and actions across the workspace

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-observability

For use in Claude.ai and ChatGPT