Slurm Node Is DRAINED: How to Find the Exact Reason

A Slurm node DRAINED state means the scheduler has intentionally prevented a compute node from receiving new jobs. This usually happens because Slurm, a health-check mechanism, or a cluster administrator identified a condition that makes the node unsuitable for additional workloads.

Seeing drain or drained in sinfo does not immediately tell you what failed. The node may have a hardware problem, a full filesystem, an invalid Slurm configuration, a failed Prolog or Epilog, or simply have been drained manually for maintenance.

Fortunately, Slurm normally stores a reason with the node state. By combining sinfo, scontrol, system logs, and a few Linux checks, an administrator can usually determine exactly why the node was drained before deciding whether it is safe to return it to service.

What Does DRAINED Mean in Slurm?

Slurm uses node states to determine whether compute resources can receive new jobs.

A node in the DRAINED state is unavailable for new job allocations. Slurm distinguishes this from DRAINING: a draining node still has one or more jobs running, but the scheduler will not assign additional jobs to it. Once the remaining jobs finish, the node transitions to DRAINED.

For example:

sinfo -N

might return:

NODELIST   NODES PARTITION STATE
node01         1   compute idle
node02         1   compute drain
node03         1   compute alloc

The important point is that DRAINED is not itself the root cause.

It is the state Slurm uses to keep the node out of normal scheduling while an underlying condition is investigated or maintenance is performed.

Start with sinfo -R

The quickest way to investigate drained nodes is:

sinfo -R

The -R option displays nodes that are unavailable together with the reason associated with their state.

Example:

REASON                     USER      TIMESTAMP           NODELIST
Low socket*core*thread      slurm     2026-08-09T08:42    node02

Or you might see something similar to:

REASON                     USER      TIMESTAMP           NODELIST
Epilog failure              root      2026-08-09T09:15    node07
tmp filesystem full         admin     2026-08-09T09:34    node12

This is often enough to determine the direction of the investigation.

sinfo can report reasons for unavailable nodes, making sinfo -R one of the most useful first commands when troubleshooting DOWN, DRAIN, or related node conditions.

If the cluster has many drained nodes, the command also provides a fast way to identify whether several systems share the same failure.

Inspect the Node with scontrol

The next command should normally be:

scontrol show node node02

A simplified output might look like:

NodeName=node02 Arch=x86_64 CoresPerSocket=16
   CPUAlloc=0 CPUEfctv=32 CPUTot=32 CPULoad=0.04
   RealMemory=128000 AllocMem=0 FreeMem=121340
   State=IDLE+DRAIN
   Partitions=compute
   BootTime=2026-08-01T07:14:32
   SlurmdStartTime=2026-08-01T07:16:05
   LastBusyTime=2026-08-09T08:38:21
   Reason=Low socket*core*thread count [slurm@2026-08-09T08:42:11]

The most important fields during the initial investigation are:

State=
Reason=

State tells you the scheduler’s current view of the node, while Reason provides the explanation recorded when the state changed.

You should also examine CPU counts, memory, configured features, GRES resources, boot time, and SlurmdStartTime. These fields can expose differences between what Slurm expects and what the operating system is actually reporting.

DRAIN Versus DOWN

DRAIN and DOWN are related but should not be treated as interchangeable.

A drained node is deliberately prevented from receiving new workloads while allowing existing jobs to finish when applicable. A node placed directly into DOWN, however, is considered unavailable, and administrative actions involving a down node can affect jobs currently associated with it. Slurm’s troubleshooting guidance specifically notes that draining can be used when administrators want existing jobs to terminate naturally before taking the system down.

This distinction matters during maintenance.

If a compute node requires work but current jobs are healthy, an administrator may intentionally run:

scontrol update NodeName=node02 State=DRAIN Reason="scheduled maintenance"

If jobs are still running, the node becomes DRAINING.

Once they finish, it becomes DRAINED.

So not every drained node represents an unexpected failure.

Configuration Mismatches

One common reason for a node to become unavailable is a mismatch between the hardware resources reported by slurmd and those configured for the node in slurm.conf.

Check the expected configuration:

scontrol show config

and:

scontrol show node node02

On the compute node, you can also use:

slurmd -C

slurmd -C prints hardware information in a format that can help when building or validating a NodeName definition.

For example, you might discover that Slurm expects:

CPUs=64 RealMemory=250000

while the node currently reports fewer usable CPUs or memory.

Possible causes include BIOS changes, disabled CPU cores, hardware replacement, VM configuration changes, or an incorrect slurm.conf.

Important: Do not simply resume the node until you understand why the reported resources differ.

Check slurmd on the Compute Node

If the configuration looks correct, verify the Slurm node daemon:

systemctl status slurmd

Then check recent messages:

journalctl -u slurmd

For the current boot:

journalctl -u slurmd -b

Depending on the cluster configuration, slurmd may also write to a dedicated log file.

For example:

grep -iE "error|drain|fail" /var/log/slurm/slurmd.log

Note: The exact path depends on how Slurm logging is configured.

Look for messages related to registration failures, memory or CPU mismatches, GRES configuration, cgroups, filesystem problems, communication failures, Prolog or Epilog execution, and authentication issues.

Valuable Tip: The timestamp shown in the node’s Reason field is particularly valuable here. Use it to focus your log investigation around the moment when the node entered the drained state.

Prolog and Epilog Failures

Slurm can automatically drain a node when job lifecycle scripts fail.

Sites often configure Prolog scripts to prepare a node before a job starts and Epilog scripts to perform cleanup after a job finishes.

If a configured Prolog returns a non-zero exit code, Slurm can place the node into DRAIN and requeue the affected job. A failing Epilog can also cause the node to be drained.

If sinfo -R reports something similar to:

Prolog failure

or:

Epilog failure

find the scripts configured on the cluster:

scontrol show config | grep -iE "Prolog|Epilog"

Then test the underlying problem instead of immediately resuming the node.

Common issues include:

missing mount points
permission errors
unavailable network storage
failed cleanup operations
missing commands
incorrect environment assumptions

Important: A resume without fixing the script may simply cause the node to drain again when the next job arrives.

Node Health Checks

Many HPC environments perform additional health checks outside normal Slurm scheduling.

Slurm supports a HealthCheckProgram that can execute periodically on compute nodes. The script itself can detect a problem and explicitly drain the node using scontrol. The Slurm configuration documentation even uses a full temporary filesystem as an example of a condition that could result in a node being drained.

Check whether one is configured:

scontrol show config | grep -i HealthCheck

If a health-check tool or custom script is involved, inspect its logs and configuration.

Typical checks may include:

filesystem availability
disk utilization
GPU health
network interfaces
memory errors
temperature
required mounts
local scratch space

This is important because Slurm itself may only display the reason supplied by the health-check script. The detailed diagnosis may exist elsewhere.

Check Filesystems and Local Storage

Storage problems are especially common on compute nodes.

Start with:

df -h

Then check inode utilization:

df -i

A filesystem can have free gigabytes while still being unable to create new files because all available inodes have been consumed.

Also verify expected mounts:

findmnt

If the cluster uses NFS, Lustre, BeeGFS, or another distributed filesystem, confirm that required filesystems are accessible from the affected node.

For example:

mount | grep lustre

A missing home directory, application filesystem, or scratch mount can be enough for a site health-check script to drain the server.

Investigate Hardware and Kernel Problems

If Slurm itself appears healthy, investigate the operating system.

Useful commands include:

dmesg -T | tail -100

and:

journalctl -p err -b

Look for memory errors, disk I/O failures, network interface problems, filesystem errors, PCIe faults, GPU errors, or kernel events occurring around the drain timestamp.

On GPU nodes, also verify the accelerator stack.

For NVIDIA systems, for example:

nvidia-smI

A GPU node can appear perfectly healthy from a CPU perspective while one accelerator has disappeared from the operating system or failed to initialize. If Slurm’s GRES configuration expects that GPU, the mismatch can prevent the node from returning to normal service.

Manually Drained Nodes

Sometimes the reason is much simpler.

An administrator may have intentionally executed:

scontrol update NodeName=node02 State=DRAIN Reason="memory replacement"

Slurm allows administrators to attach a custom reason when changing the node state.

Running:

scontrol show node node02

may reveal:

Reason=memory replacement [admin@2026-08-09T10:12:08]

The username and timestamp immediately tell you who initiated the change and when. Before resuming such a node, confirm that the maintenance activity has actually been completed.

Returning the Node to Service

Once the underlying problem has been fixed, the node can normally be returned to scheduling with:

scontrol update NodeName=node02 State=RESUME

Setting the node to RESUME clears its drain state and associated reason and allows it to return toward normal scheduling availability.

Verify the result:

sinfo -N -n node02

and:

scontrol show node node02

Ideally, the node should eventually show a schedulable state such as:

State=IDLE

Important: Do not treat RESUME as the fix itself. It only tells Slurm that the node may be considered for service again. If the underlying problem remains, the node may immediately or eventually return to DRAIN.

A Practical Troubleshooting Sequence

For most incidents, the investigation can follow this order:

sinfo -R

Identify the reason.

Then:

scontrol show node <node>

Inspect the complete node state and recorded message.

On the compute node:

systemctl status slurmd
journalctl -u slurmd -b
df -h
df -i
findmnt
dmesg -T | tail -100

If the reason mentions a Prolog, Epilog, health check, GRES device, or configuration mismatch, follow that path before changing the node state.

Once the actual problem has been corrected, resume the node:

scontrol update NodeName=<node> State=RESUME

and verify it again with:

sinfo -N -n <node>

To Wrap This Up

A drained node is therefore less mysterious than it first appears. The DRAINED state is Slurm’s way of keeping questionable resources away from new workloads while preserving information about what happened.

Starting with sinfo -R, reading the node’s Reason, and matching that timestamp against Slurm and operating-system logs usually provides a clear path from the scheduler symptom to the real problem.

If you are new on the HPC world, no worries about that. We have written a basic article explaining what HPC is. Click here to access the article. Additionally, if you want to build your own HPC cluster, click here to access an article about that!

Enjoy it 🙂