What data formats are supported?

Ray Data supports a wide variety of formats including Parquet (recommended for ML), CSV, JSON, NumPy, and various image formats for computer vision tasks.

Which ML frameworks does this skill integrate with?

It provides native support and easy conversion methods for PyTorch, TensorFlow, and Hugging Face, making it simple to pipe processed data directly into training loops.

Does Ray Data support GPU acceleration?

Yes, Ray Data allows you to request GPUs for map_batches operations, which is ideal for accelerating computationally expensive preprocessing steps like image or tensor transformations.

When should I use Ray Data instead of Pandas?

Use Ray Data when your dataset exceeds 1GB or your local memory capacity. It is designed for streaming execution and distributed processing, whereas Pandas is restricted to single-machine memory.

Can I use this skill for batch inference?

Absolutely. Ray Data is optimized for high-throughput batch inference, allowing you to load models once per worker and process data in parallel across a cluster.

Ray Data Scalable Processing

Name: Ray Data Scalable Processing
Author: Orchestra-Research

byOrchestra-Research

•

3,983

•

データサイエンスとML

Facilitates high-performance distributed data processing and streaming for large-scale machine learning workloads.

Ray Data is a specialized skill designed to streamline the handling of massive datasets for AI and ML engineering within the Claude Code environment. It enables seamless distributed data preprocessing, batch inference, and multi-modal data loading across CPU and GPU clusters. By integrating directly with popular frameworks like PyTorch and TensorFlow, it allows developers to build efficient, scalable ETL pipelines that handle data volumes far exceeding local memory, effectively bridging the gap between laptop-scale development and production-grade cluster execution.

主な機能

01GPU-accelerated transformations for high-speed image and tensor preprocessing

02Support for diverse data formats including Parquet, CSV, JSON, and multi-modal files

033,983 GitHub stars

04Streaming execution for processing datasets larger than available memory

05Native integration with PyTorch, TensorFlow, and Hugging Face ecosystems

06Seamless scaling from single-machine testing to distributed 100+ node clusters

ユースケース

01Preprocessing large-scale multi-modal datasets (images, audio, video) for ML training

02Executing distributed ETL tasks on datasets exceeding 100GB

03Building high-throughput batch inference pipelines for production models

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add orchestra-research/ai-research-skills ray-data

For use in Claude.ai and ChatGPT

Download Skill