simpo-training

1# SimPO - Simple Preference Optimization
2 
3## Quick start
4 
5SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model.
6 
7**Installation**:
8```bash
9# Create environment
10conda create -n simpo python=3.10 && conda activate simpo
11 
12# Install PyTorch 2.2.2
13# Visit: https://pytorch.org/get-started/locally/
14 
15# Install alignment-handbook
16git clone https://github.com/huggingface/alignment-handbook.git
17cd alignment-handbook
18python -m pip install .
19 
20# Install Flash Attention 2
21python -m pip install flash-attn --no-build-isolation
22```
23 
24**Training** (Mistral 7B):
25```bash
26ACCELERATE_LOG_LEVEL=info accelerate launch \
27  --config_file accelerate_configs/deepspeed_zero3.yaml \
28  scripts/run_simpo.py \
29  training_configs/mistral-7b-base-simpo.yaml
30```
31 
32## Common workflows
33 
34### Workflow 1: Train from base model (Mistral 7B)
35 
36**Config** (`mistral-7b-base-simpo.yaml`):
37```yaml
38# Model
39model_name_or_path: mistralai/Mistral-7B-v0.1
40torch_dtype: bfloat16
41 
42# Dataset
43dataset_mixer:
44  HuggingFaceH4/ultrafeedback_binarized: 1.0
45dataset_splits:
46  - train_prefs
47  - test_prefs
48 
49# SimPO hyperparameters
50beta: 2.0                  # Reward scaling (2.0-10.0)
51gamma_beta_ratio: 0.5       # Target margin (0-1)
52loss_type: sigmoid          # sigmoid or hinge
53sft_weight: 0.0             # Optional SFT regularization
54 
55# Training
56learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
57num_train_epochs: 1
58per_device_train_batch_size: 1
59gradient_accumulation_steps: 8
60…