hyperpod-slurm-debugger
Agent skill from
Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down/drain, "Node unexpectedly rebooted" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES/GPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot/Replace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on "drain before reboot", "diagnose a Slurm node", "investigate stuck jobs."
How to install hyperpod-slurm-debugger
npx skills add awslabs/agent-plugins# or add only this skill to Claude Code: copy the folder
plugins/sagemaker-ai/skills/hyperpod-slurm-debugger/
# into
~/.claude/skills/hyperpod-slurm-debugger/ (all projects)
.claude/skills/hyperpod-slurm-debugger/ (one project)
1# HyperPod Slurm Debugger23Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on4HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.5For remediation, link to the official AWS or Slurm documentation.67## When to invoke89Invoke when the user reports any of the symptoms in the [decision table](#decision-table).1011## When NOT to invoke1213- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.14- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.15- NCCL training-hang investigation — invoke `hyperpod-nccl`.16- Node unreachable via SSM — invoke `hyperpod-ssm`.1718## Constraints1920- Read-only. Do not run, recommend, or print state-mutating commands.21- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.22- IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation23 drifts the live state from the IaC plan.2425Canonical recovery URLs:26[references/slurm-details.md → Authoritative recovery documentation](references/slurm-details.md).2728## Prerequisites2930- AWS CLI v2, authenticated for the target account and region with permissions:31 - `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`32 - `ssm:StartSession` on the HyperPod-created SSM document33- [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)34 installed locally.35…
Excerpt shown under the repository's Apache-2.0 license.