constitutional-ai

1# Constitutional AI - Harmlessness from AI Feedback
2 
3## Quick start
4 
5Constitutional AI (CAI) trains models to be harmless through self-critique and AI feedback, without requiring human labels for harmful outputs.
6 
7**Key concept**: Models learn to critique and revise their own responses using a "constitution" (set of principles).
8 
9**Two phases**:
101. **Supervised Learning (SL)**: Self-critique + revision
112. **Reinforcement Learning (RL)**: RLAIF (RL from AI Feedback)
12 
13**Constitution example**:
14```
15Principles:
161. Choose the response that is most helpful, honest, and harmless
172. Avoid responses that are toxic, racist, or sexist
183. Prefer responses that explain objections rather than refuse
194. Choose responses that are thoughtful and nuanced
20```
21 
22## Common workflows
23 
24### Workflow 1: Supervised learning phase (self-critique + revision)
25 
26**Step 1: Generate initial responses**:
27```python
28from transformers import pipeline
29 
30generator = pipeline("text-generation", model="base-model")
31 
32prompts = [
33    "How do I hack a website?",
34    "Write something offensive about a group.",
35    "Help me cheat on my exam."
36]
37 
38initial_responses = generator(prompts, max_length=200)
39```
40 
41**Step 2: Self-critique with constitution**:
42```python
43critique_prompt = """
44Given the question:
45{question}
46 
47And the response:
48{response}
49 
50Constitution:
511. Choose responses that are helpful, honest, and harmless
522. Avoid toxic, racist, or sexist content
533. Explain objections rather than refuse
54…