02-rl-reward

1# RL Reward Construction with OpenJudge
2 
3Build reward signals for reinforcement learning from human feedback (RLHF) and
4reinforcement learning from AI feedback (RLAIF) using the `openjudge` library.
5 
6## When to Use This Skill
7 
8- Building scalar rewards for GRPO / REINFORCE rollout scoring
9- Generating (chosen, rejected) preference pairs for DPO / IPO
10- Best-of-N candidate selection
11- Multi-dimensional reward shaping (correctness + safety + format)
12- Replacing or bootstrapping a reward model with LLM-as-judge
13 
14## Step 1 — Choose Your Reward Strategy
15 
16Use this decision tree **before** writing any code:
17 
18```
19RL Algorithm + Task type?
20│
21├── GRPO / REINFORCE — Verifiable task (math, code, structured output)
22│   └── → POINTWISE  ✅  (FunctionGrader, exact score, zero LLM cost)
23│
24├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization)
25│   └── → PAIRWISE TOURNAMENT  ✅  (compare each rollout vs all others in group,
26│                                    reward = net win rate within group)
27│
28├── DPO / IPO / SLiC — need (chosen, rejected) pairs
29│   └── → PAIRWISE  ✅  (two-way comparison, return winner/loser)
30│
31└── Best-of-N / reranking — rank N candidates
32    └── → LISTWISE  ✅  (single call ranks all N at once)
33```
34 
35```
36Cost constraint?
37├── Low budget
38│   └── FunctionGrader (free) → pointwise; or pairwise with small judge model
39│
40├── Medium budget
41│   └── Pointwise: 2–3 LLM graders + WeightedSumAggregator
42…