What is the recommended partition size according to this skill?

The skill suggests targeting an optimal partition size of 128MB to 256MB to balance parallelism with task scheduling overhead.

Does it support PySpark specifically?

Yes, the skill contains comprehensive PySpark patterns and configuration templates for production-grade data processing.

How does this skill help with Spark OutOfMemory errors?

It provides specific memory tuning patterns for executor memory, memory overhead, and storage fractions to prevent OOM issues and manage GC pressure.

Does it provide guidance on specific file formats?

Yes, it covers optimization for Parquet and Delta Lake, including predicate pushdown, column pruning, and Z-ordering strategies.

Can this skill help with uneven data distribution?

Yes, it includes techniques for handling data skew, such as manual salting and configuring Adaptive Query Execution (AQE) skew join optimizations.

Spark Optimization

Name: Spark Optimization
Author: Microck

byMicrock

•

Data Science & ML

Optimizes Apache Spark jobs through advanced partitioning, memory management, and shuffle tuning strategies.

The Spark Optimization skill equips Claude with deep expertise for improving the performance, scalability, and cost-efficiency of Apache Spark data processing pipelines. It provides production-ready patterns for sophisticated join optimizations, memory and executor tuning, and shuffle reduction. This skill is indispensable for data engineers who need to debug slow jobs, resolve OutOfMemory (OOM) errors, handle severe data skew, or scale large-scale ETL pipelines to process massive datasets while minimizing resource consumption.

Key Features

01Shuffle optimization via Adaptive Query Execution (AQE) and pre-aggregation patterns.

02Comprehensive memory tuning for JVM, executor overhead, and storage-to-execution fractions.

03Join optimization techniques including Broadcast, Sort-Merge, and automated Skew Join handling.

04Storage format performance tuning for Parquet and Delta Lake including Z-ordering and bin-packing.

05Advanced partitioning strategies including optimal sizing and repartitioning vs. coalesce logic.

062 GitHub stars

Use Cases

01Reducing execution time and cloud infrastructure costs for high-volume data pipelines.

02Optimizing data distribution and query speed for multi-terabyte data lakes.

03Resolving recurring executor failures and memory pressure issues in production Spark clusters.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add microck/opendots-microck spark-optimization

For use in Claude.ai and ChatGPT

Key Features

01Shuffle optimization via Adaptive Query Execution (AQE) and pre-aggregation patterns.

02Comprehensive memory tuning for JVM, executor overhead, and storage-to-execution fractions.

03Join optimization techniques including Broadcast, Sort-Merge, and automated Skew Join handling.

04Storage format performance tuning for Parquet and Delta Lake including Z-ordering and bin-packing.

05Advanced partitioning strategies including optimal sizing and repartitioning vs. coalesce logic.

062 GitHub stars

Use Cases

01Reducing execution time and cloud infrastructure costs for high-volume data pipelines.

02Optimizing data distribution and query speed for multi-terabyte data lakes.

03Resolving recurring executor failures and memory pressure issues in production Spark clusters.