Scales Python data science workflows using parallel and distributed computing for larger-than-memory datasets.
This skill provides Claude with the specialized expertise to implement Dask for high-performance computing and large-scale data manipulation. It enables the processing of massive datasets that exceed available RAM by leveraging parallel DataFrames, Arrays, and Bags. Whether managing multi-file CSV processing, complex task graphs, or distributed workloads across clusters, this skill offers best practices for memory management, performance optimization, and scheduler selection to accelerate scientific discovery and data engineering pipelines.
主な機能
01Handles larger-than-RAM execution on single or distributed machines
02Includes performance optimization and memory management patterns
03Parallelizes pandas and NumPy operations for massive datasets
04Provides fine-grained control with task-based Futures for custom workflows
05Manages unstructured data processing via memory-efficient Dask Bags
06324 GitHub stars
ユースケース
01Scaling data analysis from local pandas prototypes to terabyte-scale production
02Efficiently processing thousands of log files or JSON records in parallel ETL pipelines
03Accelerating scientific research involving large-scale multidimensional arrays and Zarr/HDF5 formats
