Streamlines the development, testing, and deployment of large-scale Ray Data jobs for distributed ML workloads.
The Ray Data Development skill provides specialized guidance for building high-performance data processing pipelines using Ray Data. It supports the entire software lifecycle, from local development on A10 machines and testing on high-performance Slurm clusters (H100/H200) to production deployment on RayCluster and automation via Airflow. By offering pre-configured patterns for GPU/CPU coordination, model-based processing, and vLLM batch inference, it helps developers optimize GPU utilization, manage memory efficiently, and troubleshoot common scaling issues in massive ML datasets.
主要功能
01Optimized GPU/CPU coordination patterns for high-throughput ML batch inference
02Multi-environment workflows covering Local A10, Slurm clusters, and production RayCluster
031 GitHub stars
04Pre-built implementation templates for image classification and vLLM pipelines
05Performance tuning and troubleshooting strategies for distributed data workloads
06Deployment and scheduling guides for production environments including Airflow and JuiceFS
使用场景
01Processing large-scale datasets for machine learning training or distributed batch inference
02Migrating local data processing scripts to multi-node Ray clusters for production scale
03Deploying scalable ETL pipelines that require high-performance GPU acceleration
