hyperpod-cluster-debugger

1# HyperPod Cluster Debugger
2 
3**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes).
4 
5**Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).
6 
7`scripts/diagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references/<file>.md § <section>`.
8 
9| Reference                                                                 | Open when                                                           |
10| ------------------------------------------------------------------------- | ------------------------------------------------------------------- |
11| [cluster-diagnostics-detail.md](references/cluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L)                             |
12…