The following information is largely replicated from SchedMD’s srun man page, and is the subset that is likely most relevant to most NCCS users.
The srun command honors the following environment variables, when present (these override any inline directives within your batch script, but will be overridden by those also specified on the srun command line):
Input Variables
| PMI_FANOUT | This is used exclusively with PMI (MPICH2 and MVAPICH2) and controls the fanout of data communications. The srun command sends messages to application programs (via the PMI library) and those applications may be called upon to forward that data to up to this number of additional tasks. Higher values offload work from the srun command to the applications and likely increase the vulnerability to failures. The default value is 32. |
| PMI_FANOUT_OFF_HOST | This is used exclusively with PMI (MPICH2 and MVAPICH2) and controls the fanout of data communications. The srun command sends messages to application programs (via the PMI library) and those applications may be called upon to forward that data to additional tasks. By default, srun sends one message per host and one task on that host forwards the data to other tasks on that host up to PMI_FANOUT. If PMI_FANOUT_OFF_HOST is defined, the user task may be required to forward the data to tasks on other hosts. Setting PMI_FANOUT_OFF_HOST may increase performance. Since more work is performed by the PMI library loaded by the user application, failures also can be more common and more difficult to diagnose. |
| PMI_TIME | This is used exclusively with PMI (MPICH2 and MVAPICH2) and controls how much the communications from the tasks to the srun are spread out in time in order to avoid overwhelming the srun command with work. The default value is 500 (microseconds) per task. On relatively slow processors or systems with very large processor counts (and large PMI data sets), higher values may be required. |
| SLURM_ACCOUNT | Same as -A, –account |
| SLURM_CPUS_PER_TASK | Same as -c, –cpus-per-task |
| SLURM_DISTRIBUTION | Same as -m, –distribution |
| SLURM_EXACT=1 | Same as –exact; required for parallel execution of multiple sruns within a job |
| SLURM_JOB_NAME | Same as -J, –job-name |
| SLURM_NNODES | Same as -N, –nodes |
| SLURM_NTASKS (and SLURM_NPROCS for backwards compatibility) | Same as -n, –ntasks |
| SLURM_NTASKS_PER_CORE | Same as –ntasks-per-core |
| SLURM_NTASKS_PER_NODE | Same as –ntasks-per-node |
| SLURM_QOS | Same as -q, –qos |
| SLURM_TIMELIMIT | Same as -t, –time |
and sets the following output variables, which are present in the job’s environment during execution (run ‘env | grep -e SBATCH -e SLURM’ within your job, to obtain a full list):
Output Variables
| SLURM_JOB_ID | Numeric ID of the job allocation |
| SLURM_CPUS_ON_NODE | Count of processors available to the job on this node. Note the select/linear plugin allocates entire nodes to jobs, so the value indicates the total count of CPUs on the node. For the select/cons_res plugin, this number indicates the number of cores on this node allocated to the job |
| SLURM_CPUS_PER_TASK | Number of cpus requested per task. Only set if the –cpus-per-task option is specified. |
| SLURM_DISTRIBUTION | Distribution type for the allocated jobs. Set the distribution with -m, –distribution |
| SLURM_JOB_CPUS_PER_NODE | Count of processors available to the job on this node. |
| SLURM_JOB_DEPENDENCY | Set to value of the –dependency option. |
| SLURM_JOB_NAME | Name of the job. |
| SLURM_JOB_NODELIST (and SLURM_NODELIST for backwards compatibility) | List of nodes allocated to the job. |
| SLURM_JOB_NUM_NODES (and SLURM_NNODES for backwards compatibility) | Total number of nodes in the job’s resource allocation. |
| SLURM_JOB_PARTITION | Name of the partition in which the job is running. |
| SLURM_NTASKS (and SLURM_NPROCS for backwards compatibility) | Same as -n, –ntasks |
| SLURM_PROCID | The MPI rank (or relative process ID) of the current process |
| SLURM_RESTART_COUNT | If the job has been restarted due to system failure or has been explicitly requeued, this will be sent to the number of times the job has been restarted. |
| SLURM_STEP_NODELIST | List of nodes allocated to the step. |
| SLURM_STEP_NUM_NODES | Number of nodes allocated to the step. |
| SLURM_STEP_NUM_TASKS | Number of processes in the step. |
| SLURM_STEP_TASKS_PER_NODE | Number of processes per node within the step. |
| SLURM_STEP_ID (and SLURM_STEPID for backwards compatibility) | The step ID of the current job. |
| SLURM_SUBMIT_DIR | The directory from which srun was invoked. |
| SLURM_TASKS_PER_NODE | Number of tasks to be initiated on each node. Values are comma separated and in the same order as SLURM_NODELIST. If two or more consecutive nodes are to have the same task count, that count is followed by “(x#)” where “#” is the repetition count. For example, “SLURM_TASKS_PER_NODE=2(x3),1” indicates that the first three nodes will each execute three tasks and the fourth node will execute one task. |
| SLURM_TASK_PID | The process ID of the task being started. |
For more information on sbatch, please refer to the man pages. Use the following command, after you’ve logged onto Discover:
man sbatchor
sbatch -help