distributed-llm-pretraining-torchtitan

1# TorchTitan - PyTorch Native Distributed LLM Pretraining
2 
3## Quick start
4 
5TorchTitan is PyTorch's official platform for large-scale LLM pretraining with composable 4D parallelism (FSDP2, TP, PP, CP), achieving 65%+ speedups over baselines on H100 GPUs.
6 
7**Installation**:
8```bash
9# From PyPI (stable)
10pip install torchtitan
11 
12# From source (latest features, requires PyTorch nightly)
13git clone https://github.com/pytorch/torchtitan
14cd torchtitan
15pip install -r requirements.txt
16```
17 
18**Download tokenizer**:
19```bash
20# Get HF token from https://huggingface.co/settings/tokens
21python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
22```
23 
24**Start training on 8 GPUs**:
25```bash
26CONFIG_FILE="./torchtitan/models/llama3/train_configs/llama3_8b.toml" ./run_train.sh
27```
28 
29## Common workflows
30 
31### Workflow 1: Pretrain Llama 3.1 8B on single node
32 
33Copy this checklist:
34 
35```
36Single Node Pretraining:
37- [ ] Step 1: Download tokenizer
38- [ ] Step 2: Configure training
39- [ ] Step 3: Launch training
40- [ ] Step 4: Monitor and checkpoint
41```
42 
43**Step 1: Download tokenizer**
44 
45```bash
46python scripts/download_hf_assets.py \
47  --repo_id meta-llama/Llama-3.1-8B \
48  --assets tokenizer \
49  --hf_token=YOUR_HF_TOKEN
50```
51 
52**Step 2: Configure training**
53 
54Edit or create a TOML config file:
55 
56```toml
57# llama3_8b_custom.toml
58[job]
59dump_folder = "./outputs"
60description = "Llama 3.1 8B training"
61 
62[model]
63name = "llama3"
64flavor = "8B"
65…