hyperpod-nccl

1# HyperPod NCCL Debugger
2 
3**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.
4 
5Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts/nccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references/<file>.md § <section>`. Read-only.
6 
7**Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.
8 
9---
10 
11## Workflow
12 
131. Collect cluster name, region, namespace/job (EKS), exact NCCL error string.
142. Run the diagnostic (always — the output drives everything else).
153. For every `[FAIL]` line, `Read` the referenced section.
164. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval.
17…