nanogpt

1# nanoGPT - Minimalist GPT Training
2 
3## Quick start
4 
5nanoGPT is a simplified GPT implementation designed for learning and experimentation.
6 
7**Installation**:
8```bash
9pip install torch numpy transformers datasets tiktoken wandb tqdm
10```
11 
12**Train on Shakespeare** (CPU-friendly):
13```bash
14# Prepare data
15python data/shakespeare_char/prepare.py
16 
17# Train (5 minutes on CPU)
18python train.py config/train_shakespeare_char.py
19 
20# Generate text
21python sample.py --out_dir=out-shakespeare-char
22```
23 
24**Output**:
25```
26ROMEO:
27What say'st thou? Shall I speak, and be a man?
28 
29JULIET:
30I am afeard, and yet I'll speak; for thou art
31One that hath been a man, and yet I know not
32What thou art.
33```
34 
35## Common workflows
36 
37### Workflow 1: Character-level Shakespeare
38 
39**Complete training pipeline**:
40```bash
41# Step 1: Prepare data (creates train.bin, val.bin)
42python data/shakespeare_char/prepare.py
43 
44# Step 2: Train small model
45python train.py config/train_shakespeare_char.py
46 
47# Step 3: Generate text
48python sample.py --out_dir=out-shakespeare-char
49```
50 
51**Config** (`config/train_shakespeare_char.py`):
52```python
53# Model config
54n_layer = 6          # 6 transformer layers
55n_head = 6           # 6 attention heads
56n_embd = 384         # 384-dim embeddings
57block_size = 256     # 256 char context
58 
59# Training config
60batch_size = 64
61learning_rate = 1e-3
62max_iters = 5000
63eval_interval = 500
64 
65# Hardware
66device = 'cpu'  # Or 'cuda'
67compile = False # Set True for PyTorch 2.0
68```
69…