Can I use NeMo Curator on a single machine?

Yes, it can run on a single GPU workstation using LocalCUDACluster, but it is also designed to scale across massive GPU clusters for multi-terabyte datasets.

Does it support modalities other than text?

Yes, NeMo Curator includes specialized modules for curating image, video, and audio datasets, including aesthetic scoring, NSFW detection, and ASR transcription.

Is NeMo Curator used for real-world models?

Yes, it is the same toolkit used by NVIDIA to prepare training data for models like Nemotron-4 and to curate public datasets like RedPajama v2.

What is the main advantage of NeMo Curator over CPU tools?

The primary advantage is performance; it offers up to 16x faster fuzzy deduplication and roughly 90% cost savings on large-scale datasets by leveraging NVIDIA GPU acceleration.

NeMo Curator - GPU Data Curation

Name: NeMo Curator - GPU Data Curation
Author: zechenzhangAGI

byzechenzhangAGI

•

384

•

数据科学与机器学习

Curates high-quality datasets for LLM training using GPU-accelerated deduplication, filtering, and PII redaction.

NeMo Curator is a high-performance toolkit designed to streamline the preparation of massive datasets for large language model (LLM) training. Leveraging NVIDIA GPUs and the RAPIDS ecosystem, it provides a 16x speedup over CPU-based methods for complex tasks like fuzzy deduplication, heuristic quality filtering across 30+ metrics, and PII redaction. This skill is essential for AI researchers and engineers who need to process terabytes of multi-modal data—including text, image, video, and audio—while significantly reducing processing time and total cost of ownership.

主要功能

01Comprehensive quality filtering with 30+ built-in heuristics

02Automated PII redaction and NSFW content detection

03384 GitHub stars

04Linear scaling across multi-GPU clusters using Dask and RAPIDS

05Multi-modal support for text, image, video, and audio datasets

06GPU-accelerated fuzzy and semantic deduplication (16x faster than CPU)

使用场景

01Optimizing data curation costs and speed for large-scale AI research

02Cleaning and deduplicating massive web-scale datasets like Common Crawl

03Preparing high-quality, safe training data for proprietary foundation models

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add zechenzhangagi/ai-research-skills nemo-curator

For use in Claude.ai and ChatGPT

Download Skill