How does this skill help with Spark OOM errors?

It provides specific memory tuning patterns, including executor configuration and memory fraction adjustments, to help prevent and troubleshoot OutOfMemory errors in Spark executors.

Does this skill work with PySpark?

Yes, the implementation patterns, code snippets, and configuration templates are designed for PySpark and Spark SQL environments.

Can it help with data skew issues?

Yes, it includes patterns for identifying skew and implementing solutions like salting, broadcast joins, or leveraging Adaptive Query Execution (AQE) to handle uneven data distribution.

What storage formats are covered in the optimization patterns?

The skill includes optimization techniques specifically for columnar formats like Parquet and Delta Lake, focusing on predicate pushdown, column pruning, and Z-ordering.

Spark Optimization Pro

Name: Spark Optimization Pro
Author: pur3v4d3r

bypur3v4d3r

•

Ciencia de Datos y ML

Optimizes Apache Spark data processing jobs through advanced partitioning, memory management, and shuffle tuning.

This skill provides specialized knowledge and implementation patterns for maximizing the performance of Apache Spark jobs within your codebase. It offers production-grade strategies for handling data skew, optimizing join operations, configuring executor memory, and implementing efficient caching mechanisms. It is particularly valuable for data engineers and developers who need to scale data pipelines, debug slow-running ETL processes, or reduce cloud infrastructure costs by improving Spark resource utilization and execution efficiency.

Características Principales

01Advanced partitioning and repartitioning strategies for balanced parallelism

02Detailed memory tuning and executor configuration patterns to prevent OOM errors

031 GitHub stars

04Performance monitoring utilities to identify data skew and query plan bottlenecks

05Shuffle reduction strategies and Adaptive Query Execution (AQE) configurations

06Join optimization techniques including broadcast, sort-merge, and salted joins

Casos de Uso

01Implementing optimized storage patterns for Parquet and Delta Lake environments

02Debugging and resolving performance bottlenecks in production ETL jobs

03Scaling slow-running Spark pipelines to handle multi-terabyte datasets efficiently

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add pur3v4d3r/pur3-pkb-codebase spark-optimization

For use in Claude.ai and ChatGPT

Download Skill