How does this skill help with Dask memory errors?

It provides strategies for adjusting worker memory thresholds (target, spill, pause, terminate), managing unmanaged memory, and using persist() effectively to prevent workers from crashing.

How does it handle serialization overhead?

It provides patterns for using client.scatter and Futures to prevent large Python objects from being repeatedly serialized and sent over the network to workers.

Can it help with slow Dask jobs?

Yes, it focuses on reducing communication overhead, optimizing chunk sizes (100MB-300MB target), and identifying idle gaps in the task stream using performance reports.

Does it support Dask DataFrames?

Absolutely, it includes patterns for optimized joins, efficient shuffling using indexing, and high-throughput I/O using Parquet and categorization.

Dask Performance Optimization

Name: Dask Performance Optimization
Author: tondevrel

bytondevrel

•

数据科学与机器学习

Optimizes Dask distributed computing performance through advanced cluster tuning, memory management, and task graph refinement.

This skill provides specialized guidance for scaling Python workloads using Dask, focusing on eliminating bottlenecks like "KilledWorker" errors and high network overhead. It equips Claude with the patterns needed to manage worker memory limits, optimize data serialization with client.scatter, and resolve data skew issues through efficient repartitioning. Whether you are processing terabyte-scale datasets or building custom distributed algorithms, this skill ensures your Dask clusters run with maximum throughput and minimal idle time by applying industry-standard best practices and dashboard-driven debugging techniques.

主要功能

011 GitHub stars

02Distributed task graph optimization and operation fusing

03Cluster resource tagging for hardware-specific task routing

04Efficient data serialization using scatter/gather patterns

05Memory management and KilledWorker error prevention

06Automated performance profiling and dashboard interpretation

使用场景

01Tuning worker configurations for specific CPU or GPU-bound workloads

02Debugging out-of-memory (OOM) errors in large-scale distributed dataframes

03Reducing network I/O overhead in complex parallel computations

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add tondevrel/scientific-agent-skills dask-optimization

For use in Claude.ai and ChatGPT

Download Skill