torchforge-rl-training

1# torchforge: PyTorch-Native Agentic RL Library
2 
3torchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.
4 
5## When to Use torchforge
6 
7**Choose torchforge when you need:**
8- Clean separation between RL algorithms and infrastructure
9- PyTorch-native abstractions (no Ray dependency)
10- Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)
11- Scalable training with Monarch actor system
12- Integration with TorchTitan for model parallelism
13 
14**Consider alternatives when:**
15- You need production-ready stability → use **miles** or **verl**
16- You want Megatron-native training → use **slime**
17- torchforge is experimental and APIs may change
18 
19## Key Features
20 
21- **Algorithm isolation**: Implement RL algorithms without touching infrastructure
22- **Scalability**: From single GPU to thousands via Monarch
23- **Modern stack**: TorchTitan (training), vLLM (inference), TorchStore (sync)
24- **Loss functions**: GRPO, DAPO, CISPO, GSPO, SAPO built-in
25 
26## Architecture Overview
27 
28```
29┌─────────────────────────────────────────────────────────┐
30│ Application Layer (Your Code)                           │
31│ - Define reward models, loss functions, sampling        │
32└─────────────────────┬───────────────────────────────────┘
33                      │
34┌─────────────────────▼───────────────────────────────────┐
35…