data-processing-ray-data

1# Ray Data - Scalable ML Data Processing
2 
3Distributed data processing library for ML and AI workloads.
4 
5## When to use Ray Data
6 
7**Use Ray Data when:**
8- Processing large datasets (>100GB) for ML training
9- Need distributed data preprocessing across cluster
10- Building batch inference pipelines
11- Loading multi-modal data (images, audio, video)
12- Scaling data processing from laptop to cluster
13 
14**Key features**:
15- **Streaming execution**: Process data larger than memory
16- **GPU support**: Accelerate transforms with GPUs
17- **Framework integration**: PyTorch, TensorFlow, HuggingFace
18- **Multi-modal**: Images, Parquet, CSV, JSON, audio, video
19 
20**Use alternatives instead**:
21- **Pandas**: Small data (<1GB) on single machine
22- **Dask**: Tabular data, SQL-like operations
23- **Spark**: Enterprise ETL, SQL queries
24 
25## Quick start
26 
27### Installation
28 
29```bash
30pip install -U 'ray[data]'
31```
32 
33### Load and transform data
34 
35```python
36import ray
37 
38# Read Parquet files
39ds = ray.data.read_parquet("s3://bucket/data/*.parquet")
40 
41# Transform data (lazy execution)
42ds = ds.map_batches(lambda batch: {"processed": batch["text"].str.lower()})
43 
44# Consume data
45for batch in ds.iter_batches(batch_size=100):
46    print(batch)
47```
48 
49### Integration with Ray Train
50 
51```python
52import ray
53from ray.train import ScalingConfig
54from ray.train.torch import TorchTrainer
55 
56# Create dataset
57train_ds = ray.data.read_parquet("s3://bucket/train/*.parquet")
58 
59def train_func(config):
60    # Access dataset in training
61…