nanogpt-training

1# NanoGPT Training
2 
3## Overview
4 
5Training GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:
6 
7- **GPT-124M Architecture**: Standard transformer with RoPE and modern optimizations
8- **Tokenized Datasets**: Loading pre-tokenized shards from HuggingFace Hub or local files
9- **Modern Optimizers**: Muon optimizer with Newton-Schulz orthogonalization
10- **Mixed Precision**: bfloat16 training on A100 for 2x speedup
11 
12Training options:
13- **Baseline GPT**: Standard residual connections
14- **Experimental residual variants**: Optional alternative residual schemes for stability/efficiency
15 
16## Quick Reference
17 
18| Topic | Reference |
19|-------|-----------|
20| Model Architecture | [GPT Architecture](references/gpt-architecture.md) |
21| Data Loading | [Tokenized Data](references/tokenized-data.md) |
22| Optimizers | [Optimizers](references/optimizers.md) |
23| Training Loop | [Training Loop](references/training-loop.md) |
24| Hyperparameters | [Hyperparameters](references/hyperparameters.md) |
25 
26## Installation
27 
28```bash
29pip install torch einops numpy huggingface_hub
30```
31 
32## Minimal Example
33 
34```python
35import modal
36 
37app = modal.App("gpt-training")
38 
39image = modal.Image.debian_slim(python_version="3.11").pip_install(
40    "torch", "einops", "numpy", "huggingface_hub"
41)
42 
43@app.function(gpu="A100", image=image, timeout=3600)
44def train():
45    import torch
46    from dataclasses import dataclass
47 
48    @dataclass
49    class GPTConfig:
50        block_size: int = 1024
51…