How is data integrity verified after migration?

The skill includes a validation module that compares source and target systems for row counts, schema consistency, and sample data matches.

How does it handle complex schema mapping?

It includes a dedicated schema migration script that maps problematic legacy types (like char or specific decimals) to Delta Lake-compatible Spark types automatically.

Does it support incremental data loads?

Yes, the data migration patterns include logic for batch processing and can be configured with incremental columns to handle large-scale transfers.

Which source systems are supported by this migration skill?

The skill provides patterns for migrating from on-premises Hadoop (Hive), Snowflake, AWS Redshift, Azure Synapse, and legacy RDBMS like Oracle or Teradata.

Databricks Migration Deep Dive

Name: Databricks Migration Deep Dive
Author: jeremylongshore

byjeremylongshore

•

1,243

•

Database Management

Orchestrates comprehensive platform migrations from legacy data systems like Hadoop and Snowflake into Databricks using automated assessment and Delta Lake conversion patterns.

This skill provides a robust framework for managing complex data migrations to Databricks. It guides users through the entire migration lifecycle, including automated discovery of legacy Hive metastores, intelligent schema mapping for Delta Lake compatibility, and high-performance data transfer scripts. Whether you are moving from on-premises Hadoop clusters or cloud warehouses like Redshift and Snowflake, this skill provides the implementation patterns, batch-processing logic, and validation protocols necessary to ensure a reliable transition to the Databricks Lakehouse architecture.

Key Features

01Automated source system assessment and migration prioritization logic

02Intelligent schema conversion from Hive/legacy SQL to Delta Lake types

03Batch-based data migration scripts with checkpointing and partitioning

04Tailored migration patterns for Hadoop, Snowflake, Redshift, and Synapse

05Automated validation tools for row counts, schema matching, and data integrity

061,243 GitHub stars

Use Cases

01Consolidating legacy data warehouses like Oracle or Teradata into Databricks

02Migrating data workloads and schemas from Snowflake to Delta Lake

03Modernizing on-premises Hadoop/Hive clusters to a Databricks cloud environment

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-migration-deep-dive

For use in Claude.ai and ChatGPT

Download Skill

Key Features

01Automated source system assessment and migration prioritization logic

02Intelligent schema conversion from Hive/legacy SQL to Delta Lake types

03Batch-based data migration scripts with checkpointing and partitioning

04Tailored migration patterns for Hadoop, Snowflake, Redshift, and Synapse

05Automated validation tools for row counts, schema matching, and data integrity

061,243 GitHub stars

Use Cases

01Consolidating legacy data warehouses like Oracle or Teradata into Databricks

02Migrating data workloads and schemas from Snowflake to Delta Lake

03Modernizing on-premises Hadoop/Hive clusters to a Databricks cloud environment

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add jeremylongshore/claude-code-plugins-plus-skills databricks-migration-deep-dive

For use in Claude.ai and ChatGPT

Download Skill