Slurm cheat sheet

Slurm is the job scheduler that we use in Unity. For an introduction on Slurm, see Introduction to Slurm: The Job Scheduler.

Common terms

The following is a list of common terms in Slurm:

• Node - a single computer.
• Socket - a single CPU.
• Core - a single compute unit inside a CPU.
• CPU - one core, except on power9 where it is a Thread within a Core.
• Job - a schedulable unit; an allocation of resources.
• Job step - a set of related processes within a task. .batch is the script as submitted, .0...X are any srun invocations from within the script
• Task - a process within a job step.

Submit a job with arguments

To submit a job in Slurm, use the sbatch command. If you want to set parameters for your job, there are many arguments available for you to add to your batch file. See the Introduction to batch jobs page for examples of how to create a batch file with arguments. Note that any arguments specified on the command line when submitting your job override those in the file.

The following table contains a list of arguments you can use to specify parameters for your batch job. For more detailed information on submitting jobs in Slurm, see the sbatch manpage.

Job steps

Inside of your batch file, use the srun command to specify the command to run across the nodes allocated.

It’s uncommon to need to specify other arguments with this command, but srun accepts most of the arguments from the arguments table if necessary, with the exceptions of --array and --dependency. See the srun manpage for more detailed information.

Interactive jobs

To start an interactive job, use the salloc command followed by arguments that specify details about your job.

Similar to srun, salloc takes the same arguments as sbatch, except --array. The Using SALLOC page has more information, as does the official salloc manpage.

Modify a job

It’s possible to change some job properties while they are pending, and a few after they start running with the scontrol modify jobid= command. Use <tab> completion to see all the various parameters that can be changed for pending jobs. The following is a list of the most common arguments:

You can use the separate command scontrol top <jobid_list> to give higher priority to specific jobs compared to your other jobs in the same partition. This command accepts a comma-separated list of job IDs. Note that this command only works for jobs within a single partition.

Cancel a job

To cancel running and pending jobs, use scancel jobid. To cancel a running step, use scancel jobid.step.

Check a running job

There are multiple ways to check the progress and efficiency of a running job. See Monitoring a Batch job for details.

Check on recent job status

To check on the status of recent jobs, use the squeue command followed by an argument that specifies what type of information you want to view. Note that only jobs that are currently running or finished in the last ~5 minutes are available with the squeue command. The following table shows common argument options. For more details, see the squeue manpage.

Check on older job status

To check on the status of an older job, use the sacct command followed by an argument that specifies what type of information you want to view. The following table shows common argument options. For more detailed information, see the sacct manpage.

Check on node status

To check on the status of slurm nodes and partitions, use the sinfo command followed by an argument that specifies what type of information you want to view. The following table shows common argument options. For more details, see the sinfo manpage.