fine-tuning-with-trl

1# TRL - Transformer Reinforcement Learning
2 
3## Quick start
4 
5TRL provides post-training methods for aligning language models with human preferences.
6 
7**Installation**:
8```bash
9pip install trl transformers datasets peft accelerate
10```
11 
12**Supervised Fine-Tuning** (instruction tuning):
13```python
14from trl import SFTTrainer
15 
16trainer = SFTTrainer(
17    model="Qwen/Qwen2.5-0.5B",
18    train_dataset=dataset,  # Prompt-completion pairs
19)
20trainer.train()
21```
22 
23**DPO** (align with preferences):
24```python
25from trl import DPOTrainer, DPOConfig
26 
27config = DPOConfig(output_dir="model-dpo", beta=0.1)
28trainer = DPOTrainer(
29    model=model,
30    args=config,
31    train_dataset=preference_dataset,  # chosen/rejected pairs
32    processing_class=tokenizer
33)
34trainer.train()
35```
36 
37## Common workflows
38 
39### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
40 
41Complete pipeline from base model to human-aligned model.
42 
43Copy this checklist:
44 
45```
46RLHF Training:
47- [ ] Step 1: Supervised fine-tuning (SFT)
48- [ ] Step 2: Train reward model
49- [ ] Step 3: PPO reinforcement learning
50- [ ] Step 4: Evaluate aligned model
51```
52 
53**Step 1: Supervised fine-tuning**
54 
55Train base model on instruction-following data:
56 
57```python
58from transformers import AutoModelForCausalLM, AutoTokenizer
59from trl import SFTTrainer, SFTConfig
60from datasets import load_dataset
61 
62# Load model
63model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
64tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
65…