Apache Spark Optimization

关于

This skill provides production-grade patterns for enhancing Apache Spark job performance, helping developers build scalable data pipelines. It enables Claude to implement optimal partitioning strategies, configure fine-grained executor memory settings, and debug performance bottlenecks like data skew or excessive shuffling. By applying industry-standard techniques such as Adaptive Query Execution (AQE), broadcast joins, and efficient serialization, this skill ensures that big data processing remains cost-effective and performant even at a multi-terabyte scale.

主要功能

• Executor memory and storage fraction tuning
• Join optimization (Broadcast, Bucket, and Sort-Merge)
• Adaptive Query Execution (AQE) configuration
• Advanced partitioning and data skew salting
• 23,194 GitHub stars
• Efficient data format and serialization settings

使用场景

• Scaling data pipelines to handle multi-terabyte datasets efficiently
• Reducing cloud infrastructure costs by minimizing Spark job execution time
• Resolving OutOfMemory (OOM) errors and executor failures in production