slurm_rcc

slurm_rcc, a batch job scheduler which controls job execution on the FSU Research Computing Center (RCC) computer cluster. It allows a user to set up a batch file describing how a program is to be executed in parallel. Once the batch file is submitted, it goes into a queue. waiting for a time when the desired number of processors are available, when it begins execution. On job completion, the program output is gathered and returned it to the user. Until the job is complete, the scheduler allows the user to query its current status.

A user typically logs into a special login node of the cluster, which is intended only for editing, file management, job submission, and other small interactive tasks.

The user wishes to run a parallel program on several processors of the cluster. To do so, the user must create an executable version of the program, write a suitable batch job script describing the job limits, and listing the commands to be executed, and then submit the script for processing.

The job script can be thought of as consisting of two parts:

Commands to the scheduler, describing the resources needed;
Ordinary operating system commands, which could essentially be typed interactively, which cause the program to execute, but which may also create or delete files, print the date, change directories, and so on.

The user has several separate issues when preparing a first job script:

Scheduler issues:

understanding the local "dialect" of the scheduler, and the local job policies, such as the names of queues, the limits on the number of processors that can be requested, and so on. The exact form of the scheduler commands is usually obscure, but you only have to figure it out once.

and

File and Directory issues:

The first thing to realize is that the script will essentially "log in" to your account, and (by default), will automatically move to the directory from which the script was submitted. Thus, it is a sensible plan to store your input data files and your job script in the same directory. After your job has executed, you can also expect to find your output files here. If that is not actually your wish, it is possible to indicate a different working directory at the time you submit your script, or else, you can simply insert the appropriate "cd" (change directory) command into the second part of your script.

and

Program issues:

For OpenMP, this primarily involves finding out how to load the necessary libraries, the name of the compiler to use, and the switches required to compile with OpenMP, and noting whether you want exclusive access to a node so you aren't competing for cores.
For MPI, it is often necessary to invoke a locally defined setup script, to know the names of the compiler scripts, and to know the syntax for the command to execute a program under MPI, and to determine the appropriate way to request a given number of nodes.
for (nonparallel) MATLAB, you need to understand how to use MATLAB in batch mode; that means suppressing the exuberant display of the MATLAB logo and the desktop; it means saving graphics into files using MATLAB's print() command.
for parallel MATLAB, you need to know how many parallel labs you can request, and any special commands that are necessary to get parallel MATLAB running. At the FSU RCC cluster facility, for instance, there are a limited number of parallel MATLAB licenses, and parallel MATLAB may only be run by invoking a special script.

Local Installation:

At FSU RCC, users of the cluster must apply for an account by going to the web page: https://rcc.fsu.edu, and choosing the "My Account" item from the menu on the side, and on the new page, selecting "Sign Up". Accounts are available as general access accounts (anyone) and owner-based accounts (requiring authorization from the "owner").

Any FSU faculty member can get a general access account; any researcher can also get a general access account if they have an FSU faculty sponsor.

Some researchers support the system, and in return have priority access to components of the system - they are, in essence, "owners" of part of the system. You can get an account on an owner-based component at the discretion of the owner.

Once you have applied for an account, and it has been approved, you can access the system, using an ID (which may be the same as your FSU ID, or not) and a password associated with the RCC system. To log into the system, use ssh and the address of the component to which you have been assigned. For instance, I log in using the command:

        ssh sc.rcc.fsu.edu

        ssh -Y sc.rcc.fsu.edu

to enable X window graphics, which are needed, for instance, if you want to work interactively with MATLAB.

To transfer files between a local system and the RCC, you need the sftp command. I do this in a second window, so that I have interactive access in my ssh window, while file transfers occur in the sftp window. The command that makes the connection for me is:

        sftp sc.rcc.fsu.edu

and I put files from my local system to the RCC by a command like

        put fred.txt

and get files from the RCC back to the local system by

        get jeff.txt

The main reason for using the cluster is to be able to compute in batch mode - one or many jobs, submitted to a queue, to run "eventually". You can log out after you submit jobs, and log in later at your convenience to collect the output from completed jobs. Parallel programs can be run on multiple processors this way. Matlab programs, whether parallel or sequential, can also be submitted to the batch queue.

The commands that make a job run in the batch queue form a job script. The first part of the script contains commands to the job scheduler. These commands specify the maximum time limit, the number of processes, the particular queue you will use, and so on. While general users might use the "classroom" queue, I access the queue "gunzburg_q" associated with my research group.

After the scheduler commands come a sequence of commands that you might imagine typing in interactively; that is, these might be the normal sequence of UNIX commands you would issue to run a particular job.

Briefly, if you have a job script file called fred.sh you can first determine what partitions are available on the system by

        sinfo

Then you can submit your job by a command like

        sbatch fred.sh

or submit to a specific partition by:

        sbatch -p myqueue fred.sh

You can check to see the status of all jobs by the command

        squeue

or the status of your jobs by the command

        squeue -u jburkardt

and, seeing that your job has the identifying number 1750, you could cancel your job by

        scancel 1750

or, if your job runs to completion,you should find an output file in your RCC directory containing the output, or the error messages that explain why you didn't actually get any output.

Licensing:

The information on this web page is distributed under the MIT license.

Related Data and Programs:

matlab_commandline, a MATLAB code which illustrate how MATLAB can be run from the UNIX command line, that is, not with the usual interactive command window.

matlab_compiler, MATLAB programs which illustrate the use of the Matlab compiler, which allows you to run a Matlab application outside the Matlab environment.

matlab_parallel, examples which illustrate local parallel programming on a single computer with MATLAB's Parallel Computing Toolbox.

moab, examples which illustrate the use of the MOAB job scheduler for batch execution of jobs on a computer cluster.

mpi, a C code which uses the MPI application program interface for carrying out parallel computations in a distributed memory environment.

openmp, a Fortran90 code which uses the OpenMP application program interface for carrying out parallel computations in a shared memory environment.

slurm_h2p, examples which demonstrate the use of the SLURM batch job scheduler for the h2p computer cluster, as administered by the Center for Research Computing (CRC) at the University of Pittsburgh.

Source Code:

ENVIRON is a batch job script that simply queries the values of certain environment variables.

environ.sh, a job script.
environ_output.txt, the output file.

HELLO is a batch job script that compiled and rund a program. The script also "cleans up" after itself, that is, it discards the executable program once the job is complete.

hello.sh, a job script.
hello_output.txt, the output file.

HELLO_OPENMP illustrates the compilation and execution of a program that includes OpenMP directives.

hello_openmp.sh, a job script.
hello_openmp_output.txt, the output file.

HELLO_MPI illustrates the compilation and execution of a program that includes MPI directives.

hello_mpi.sh, a job script.
hello_mpi_output.txt, the output file.

JOB_ARRAY illustrates how a single SLURM sbatch command can submit an array of jobs, which use the same batch file, but which are each given a different task id. In this example, the batch file was invoked using the command

        sbatch -a 0-3 job_array.sh

meaning that the job_array.sh file was essentially submitted 4 times, with the environment variable $SLURM_ARRAY_TASK_ID set to 0, 1, 2 and 3.

job_array.sh, a job script.
job_array_0_output.txt, the output file from the job with task ID 0.
job_array_1_output.txt, the output file from the job with task ID 1.
job_array_2_output.txt, the output file from the job with task ID 2.
job_array_3_output.txt, the output file from the job with task ID 3.

POWER_TABLE shows how a MATLAB program can be run through the scheduler. We prepare a file of input commands, and invoke MATLAB on the commandline.

power_table.sh, a job script.
power_table.m, a MATLAB M file to be executed.
power_table_output.txt, the output file.

Last revised on 30 September 2024.