huggingface-accelerate

1# HuggingFace Accelerate - Unified Distributed Training
2 
3## Quick start
4 
5Accelerate simplifies distributed training to 4 lines of code.
6 
7**Installation**:
8```bash
9pip install accelerate
10```
11 
12**Convert PyTorch script** (4 lines):
13```python
14import torch
15+ from accelerate import Accelerator
16 
17+ accelerator = Accelerator()
18 
19  model = torch.nn.Transformer()
20  optimizer = torch.optim.Adam(model.parameters())
21  dataloader = torch.utils.data.DataLoader(dataset)
22 
23+ model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)
24 
25  for batch in dataloader:
26      optimizer.zero_grad()
27      loss = model(batch)
28-     loss.backward()
29+     accelerator.backward(loss)
30      optimizer.step()
31```
32 
33**Run** (single command):
34```bash
35accelerate launch train.py
36```
37 
38## Common workflows
39 
40### Workflow 1: From single GPU to multi-GPU
41 
42**Original script**:
43```python
44# train.py
45import torch
46 
47model = torch.nn.Linear(10, 2).to('cuda')
48optimizer = torch.optim.Adam(model.parameters())
49dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
50 
51for epoch in range(10):
52    for batch in dataloader:
53        batch = batch.to('cuda')
54        optimizer.zero_grad()
55        loss = model(batch).mean()
56        loss.backward()
57        optimizer.step()
58```
59 
60**With Accelerate** (4 lines added):
61```python
62# train.py
63import torch
64from accelerate import Accelerator  # +1
65 
66accelerator = Accelerator()  # +2
67 
68model = torch.nn.Linear(10, 2)
69optimizer = torch.optim.Adam(model.parameters())
70…