Does it support mock data generation?

Absolutely, the skill includes factory patterns and data builders to generate realistic, schema-compliant test datasets for various entities like customers and orders.

How does this skill help with Spark testing?

It provides pre-configured pytest fixtures and helper functions to manage Spark sessions and validate DataFrame contents, schema, and uniqueness efficiently.

Can I use this for Delta Live Tables (DLT)?

Yes, it includes specific implementation patterns for testing DLT transformation logic and expectations outside of the DLT runtime environment.

Is it compatible with standard CI/CD tools?

Yes, these patterns are built on top of pytest and standard Spark APIs, making them compatible with any CI/CD runner that supports Python environments.

Databricks Testing Patterns

Name: Databricks Testing Patterns
Author: vivekgana

byvivekgana

Seguridad y Pruebas

Provides standardized pytest fixtures, Spark DataFrame assertion utilities, and integration testing patterns for Databricks data pipelines.

Acerca de

This skill streamlines the testing lifecycle for Spark applications by offering a comprehensive suite of reusable patterns and utilities for Claude. It includes session-scoped Spark fixtures for pytest, custom DataFrame schema and content assertions, and robust mock data generation factories. Whether you are unit testing individual transformations or performing end-to-end integration tests on Delta Live Tables (DLT) and Workspace workflows, this skill helps ensure data quality and pipeline reliability through automated, CI/CD-ready testing strategies.

Características Principales

Reusable Spark session fixtures for optimized pytest execution
Custom DataFrame schema and data integrity assertion helpers
0 GitHub stars
Automated mock data generation factories for complex schemas
Integration testing patterns for Delta Live Tables (DLT) and pipelines
Parameterized testing support for verifying multiple data scenarios

Casos de Uso

Validating Spark transformation logic before deploying to production environments
Generating realistic synthetic data to test edge cases and boundary conditions in pipelines
Implementing automated quality gates in CI/CD workflows for Databricks engineering

Acerca de

Características Principales

Reusable Spark session fixtures for optimized pytest execution
Custom DataFrame schema and data integrity assertion helpers
0 GitHub stars
Automated mock data generation factories for complex schemas
Integration testing patterns for Delta Live Tables (DLT) and pipelines
Parameterized testing support for verifying multiple data scenarios

Casos de Uso

Validating Spark transformation logic before deploying to production environments
Generating realistic synthetic data to test edge cases and boundary conditions in pipelines
Implementing automated quality gates in CI/CD workflows for Databricks engineering