Accelerates LLM data curation using GPU-powered deduplication, quality filtering, and PII redaction at scale.
NeMo Curator is a specialized toolkit designed to prepare high-quality training datasets for large language models. By leveraging NVIDIA RAPIDS and GPU acceleration, it performs complex operations like fuzzy deduplication up to 16x faster than CPU-based alternatives. It provides a comprehensive pipeline for text, image, video, and audio data, including heuristic filtering, semantic deduplication, and PII redaction, making it essential for researchers and engineers building production-grade AI models from massive, unrefined corpora.
主な機能
01GPU-accelerated fuzzy and semantic deduplication (16x faster than CPU)
02Advanced quality filtering with over 30 heuristic and classifier-based filters
033,983 GitHub stars
04Automated PII redaction and NSFW content detection for safe training
05Multimodal support for text, image, video, and audio datasets
06Distributed processing across GPU clusters using Dask and RAPIDS
ユースケース
01Cleaning and deduplicating multimodal datasets containing images and video
02Redacting sensitive personal information from large-scale corporate data repositories
03Preparing massive web-scraped datasets like Common Crawl for LLM pre-training
