Optimizes Apache Spark jobs using advanced partitioning, memory management, and shuffle tuning patterns.
The Spark Optimization skill provides Claude with production-ready patterns and configurations to enhance the performance and scalability of data engineering pipelines. It equips the assistant with deep technical expertise in Spark's execution model, enabling it to resolve common bottlenecks like data skew, excessive shuffles, and memory overhead. Whether you are debugging a failing job or fine-tuning a large-scale ETL process, this skill offers actionable code snippets and configuration strategies for partitioning, broadcast joins, and caching to ensure cost-efficient and high-speed data processing.
主な機能
01Advanced partitioning and repartitioning strategies for balanced workloads
02Data format optimization for Parquet and Delta Lake storage
03Detailed memory management and executor configuration templates
040 GitHub stars
05Join optimization techniques including broadcast, bucket, and skew join handling
06Shuffle reduction patterns to minimize network and disk I/O
ユースケース
01Fixing Out-of-Memory (OOM) errors and executor failures in large Spark jobs
02Implementing Adaptive Query Execution (AQE) and data salting to handle skewed datasets
03Accelerating slow ETL pipelines and reducing cloud compute costs
