openrlhf-training

1# OpenRLHF - High-Performance RLHF Training
2 
3## Quick start
4 
5OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.
6 
7**Installation**:
8```bash
9# Launch Docker container
10docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \
11  -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:25.02-py3 bash
12 
13# Uninstall conflicts
14sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y
15 
16# Install OpenRLHF with vLLM
17pip install openrlhf[vllm]
18```
19 
20**PPO Training** (Hybrid Engine):
21```bash
22ray start --head --node-ip-address 0.0.0.0 --num-gpus 8
23 
24ray job submit --address="http://127.0.0.1:8265" \
25  --runtime-env-json='{"working_dir": "/openrlhf"}' \
26  -- python3 -m openrlhf.cli.train_ppo_ray \
27  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \
28  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \
29  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \
30  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \
31  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \
32  --colocate_all_models \
33  --vllm_gpu_memory_utilization 0.5 \
34  --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
35  --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \
36  --save_path ./output/llama3-8b-rlhf \
37  --micro_train_batch_size 8 --train_batch_size 128 \
38  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \
39  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \
40  --zero_stage 3 --bf16 \
41  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \
42…