distributed-training-megatron-core

1# Megatron-Core - Large-Scale LLM Training
2 
3## Quick start
4 
5Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.
6 
7**Installation**:
8```bash
9# Docker (recommended)
10docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:25.04-py3
11 
12# Or pip
13pip install megatron-core
14```
15 
16**Simple distributed training**:
17```bash
18# Train with 2 GPUs using data parallelism
19torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py
20 
21# Or LLaMA-3 8B training
22./examples/llama/train_llama3_8b_fp8.sh
23```
24 
25## Common workflows
26 
27### Workflow 1: Train LLaMA-style model with 3D parallelism
28 
29Copy this checklist:
30 
31```
32LLaMA Training Setup:
33- [ ] Step 1: Choose parallelism configuration
34- [ ] Step 2: Configure training hyperparameters
35- [ ] Step 3: Launch distributed training
36- [ ] Step 4: Monitor performance metrics
37```
38 
39**Step 1: Choose parallelism configuration**
40 
41Model size determines parallelism strategy:
42 
43| Model Size | GPUs | Tensor Parallel | Pipeline Parallel | Data Parallel | Context Parallel |
44|------------|------|-----------------|-------------------|---------------|------------------|
45| 7B | 8 | 1 | 1 | 8 | 1 |
46| 13B | 8 | 2 | 1 | 4 | 1 |
47| 70B | 64 | 4 | 4 | 4 | 1 |
48| 405B | 128 | 8 | 8 | 2 | 2 |
49 
50**Step 2: Configure training hyperparameters**
51 
52```bash
53#!/bin/bash
54# train_llama_70b.sh
55 
56GPUS_PER_NODE=8
57NNODES=8  # 64 GPUs total
58TP=4      # Tensor parallel
59PP=4      # Pipeline parallel
60…