data-processing-nemo-curator

1# NeMo Curator - GPU-Accelerated Data Curation
2 
3NVIDIA's toolkit for preparing high-quality training data for LLMs.
4 
5## When to use NeMo Curator
6 
7**Use NeMo Curator when:**
8- Preparing LLM training data from web scrapes (Common Crawl)
9- Need fast deduplication (16× faster than CPU)
10- Curating multi-modal datasets (text, images, video, audio)
11- Filtering low-quality or toxic content
12- Scaling data processing across GPU cluster
13 
14**Performance**:
15- **16× faster** fuzzy deduplication (8TB RedPajama v2)
16- **40% lower TCO** vs CPU alternatives
17- **Near-linear scaling** across GPU nodes
18 
19**Use alternatives instead**:
20- **datatrove**: CPU-based, open-source data processing
21- **dolma**: Allen AI's data toolkit
22- **Ray Data**: General ML data processing (no curation focus)
23 
24## Quick start
25 
26### Installation
27 
28```bash
29# Text curation (CUDA 12)
30uv pip install "nemo-curator[text_cuda12]"
31 
32# All modalities
33uv pip install "nemo-curator[all_cuda12]"
34 
35# CPU-only (slower)
36uv pip install "nemo-curator[cpu]"
37```
38 
39### Basic text curation pipeline
40 
41```python
42from nemo_curator import ScoreFilter, Modify
43from nemo_curator.datasets import DocumentDataset
44import pandas as pd
45 
46# Load data
47df = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})
48dataset = DocumentDataset(df)
49 
50# Quality filtering
51def quality_score(doc):
52    return len(doc["text"].split()) > 5  # Filter short docs
53 
54filtered = ScoreFilter(quality_score)(dataset)
55 
56# Deduplication
57…