How does this skill help with Spark OOM errors?

It provides memory tuning patterns, including executor configuration and memory fraction adjustments, to prevent memory pressure and spill-to-disk issues during execution.

Can this skill assist with data skew?

Yes, it includes specific patterns for manual salting joins and enabling Adaptive Query Execution (AQE) settings to handle uneven data distribution across partitions.

Does it support different storage formats?

It covers optimization for high-performance formats like Parquet and Delta Lake, focusing on predicate pushdown, column pruning, and Z-ordering.

Is this specific to PySpark?

While the code examples are provided in PySpark, the core architectural concepts of partitioning, shuffle optimization, and memory management apply to all Spark-supported languages.

Spark Optimization

Name: Spark Optimization
Author: as4584

byas4584

데이터 과학 및 ML

Optimizes Apache Spark performance through advanced partitioning, memory tuning, and shuffle management strategies.

소개

This skill provides comprehensive guidance for diagnosing and resolving performance bottlenecks in Apache Spark applications. It offers production-ready patterns for efficient memory management, join optimizations (including broadcast and salt joins), data skew mitigation, and storage format tuning. Whether you are dealing with OutOfMemory (OOM) errors, slow shuffles, or scaling data pipelines for massive datasets, this skill equips Claude with the technical patterns needed to build robust, high-performance distributed data processing jobs.

주요 기능

0 GitHub stars
Shuffle reduction and data format optimization
Caching and persistence management
Partitioning strategies for balanced parallelism
Advanced join optimization and skew handling
Memory and executor configuration tuning

사용 사례

Debugging OutOfMemory (OOM) errors in large-scale Spark jobs
Implementing bucketed joins to eliminate expensive shuffles
Improving the execution time of slow data processing pipelines

소개

주요 기능

0 GitHub stars
Shuffle reduction and data format optimization
Caching and persistence management
Partitioning strategies for balanced parallelism
Advanced join optimization and skew handling
Memory and executor configuration tuning

사용 사례

Debugging OutOfMemory (OOM) errors in large-scale Spark jobs
Implementing bucketed joins to eliminate expensive shuffles
Improving the execution time of slow data processing pipelines