How does this skill handle HTTP 429 'Too Many Requests' errors?

It implements an exponential backoff strategy with random jitter, which progressively increases wait times between retries while preventing multiple clients from hitting the API at the exact same time.

What are the common Databricks API rate limits?

Databricks typical limits include 30 req/s for Workspace APIs, 10 req/s for SQL Warehouses, and 100 req/s for Unity Catalog, though these can vary by workspace tier.

Can I use this for bulk operations?

Yes, the skill includes async batch processing patterns that use semaphores to control concurrency, making it ideal for bulk cluster or job management.

Why is idempotency important for Databricks job runs?

Idempotency ensures that if a network error occurs during a job submission, retrying the request won't result in duplicate job executions, saving compute costs and preventing data corruption.

Databricks API Resilience & Rate Limiting

Name: Databricks API Resilience & Rate Limiting
Author: jeremylongshore

byjeremylongshore

•

1,243

•

Cloud Infrastructure

Optimizes Databricks API interactions using exponential backoff, request queuing, and idempotency patterns to ensure reliable workspace automation.

This skill provides a comprehensive framework for managing Databricks API consumption, which is essential for high-scale automation and production orchestration. It implements sophisticated retry logic with exponential backoff and jitter, token bucket rate limiting tailored to specific Databricks tiers (Workspace, SQL, and Unity Catalog), and idempotent job submission patterns. By integrating these production-grade patterns, developers can perform bulk cluster updates, orchestrate complex data migrations, and run high-concurrency SQL queries without encountering disruptive 429 throttling errors or duplicate executions.

Key Features

01Tiered token bucket rate limiting for Workspace, SQL Warehouse, and Unity Catalog APIs

02Usage monitoring utilities to track and throttle API consumption proactively

03Asynchronous batch processing with configurable concurrency and semaphore control

041,243 GitHub stars

05Deterministic idempotency tokens for safe job submission and retry logic

06Exponential backoff with jitter for automatic handling of 429 and 503 errors

Use Cases

01Performing bulk configuration updates across hundreds of clusters without workspace throttling

02Building reliable CI/CD pipelines that manage thousands of concurrent automated job runs

03Scaling Unity Catalog object migrations and permission grants in high-volume environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-rate-limits

For use in Claude.ai and ChatGPT

Download Skill

Key Features

01Tiered token bucket rate limiting for Workspace, SQL Warehouse, and Unity Catalog APIs

02Usage monitoring utilities to track and throttle API consumption proactively

03Asynchronous batch processing with configurable concurrency and semaphore control

041,243 GitHub stars

05Deterministic idempotency tokens for safe job submission and retry logic

06Exponential backoff with jitter for automatic handling of 429 and 503 errors

Use Cases

01Performing bulk configuration updates across hundreds of clusters without workspace throttling

02Building reliable CI/CD pipelines that manage thousands of concurrent automated job runs

03Scaling Unity Catalog object migrations and permission grants in high-volume environments

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-rate-limits

For use in Claude.ai and ChatGPT

Download Skill