pbs_psc

pbs_psc, examples which illustrate the use of the Portable Batch System (PBS), a scheduler which controls the submission and execution of jobs on the Pittsburgh Supercomputing Center (PSC) computer clusters.

A user typically logs into a special login node of the cluster, which is intended only for editing, file management, job submission, and other small interactive tasks.

The user wishes to run a parallel program on several processors of the cluster. To do so, the user must create an executable version of the program, write a suitable PBS batch job script describing the job limits, and listing the commands to be executed, and then submit the script to for processing using the qsub command.

The job script can be thought of as consisting of two parts:

Commands to PBS, which must begin with the string #PBS, and which describe properties of the job;
Ordinary operating system commands, which could essentially be typed interactively, which cause the program to execute, but which may also create or delete files, print the date, change directories, and so on.

The user has several separate issues when preparing a first job script:

PBS issues:

understanding the local "dialect" of PBS, and the local job policies, such as the names of queues, the limits on the number of processors that can be requested, and so on. The exact form of the PBS commands is usually obscure, but you only have to figure it out once.

and

File and Directory issues:

The first thing to realize is that the PBS script will essentially "log in" to your home directory, and thus, will be able to access files in that home directory easily. The job will also have executed your noninteractive customization commands (which might be in .bashrc, for instance), so that you may be able to take advantange of some of the shortcuts you have defined there. If that's not where you want to be, you can issue cd commands to move to another directory. But a very convenient command is
```
            cd $PBS_O_WORKDIR
          
```
which changes to the exact directory from which you submitted the script. This is a very natural thing to do.

and

Program issues:

For OpenMP, this primarily involves finding out the name of the compiler, and the switches required to compile with OpenMP;
for MPI, it is often necessary to invoke a locally defined setup script, to know the names of the compiler scripts, and to know the syntax for the command to execute a program under MPI.
for (nonparallel) MATLAB, you need to understand how to use MATLAB in batch mode; that means suppressing the exuberant display of the MATLAB logo and the desktop; it means saving graphics into files using MATLAB's print() command.
for parallel MATLAB, you need to know how many parallel labs you can request, and any special commands that are necessary to get parallel MATLAB running. There will be a limited number of parallel MATLAB licenses.

To log into the system, use ssh and the address of the component to which you have been assigned. For instance, you might log into the PSC Greenfield system by inserting your username in the following command:

        ssh USERNAME@greenfield.psc.xsede.org

To transfer files, you need the sftp command. I do this in a second window, so that I have interactive access in my ssh window, while file transfers occur in the sftp window. The command that makes the connection for me is:

        sftp USERNAME@greenfield.psc.xsede.org

and I put files from my local system to the PSC by a command like

        put fred.txt

and get files from the PSC back to the local system by

        get jeff.txt

The main reason for using a cluster is to be able to compute in batch mode - one or many jobs, submitted to a queue, to run "eventually". You can log out after you submit jobs, and log in later at your convenience to collect the output from completed jobs. Parallel programs can be run on multiple processors this way. Matlab programs, whether parallel or sequential, can also be submitted to the batch queue.

The commands that make a job run in the batch queue form a PBS script. The first part of the script contains commands to the job scheduler. These commands begin with the string "#PBS" and specify the maximum time limit, the number of processes, the particular queue you will use, and so on. Most batch jobs should go into the "batch" queue.

The PBS command "#PBS -l nodes=1:ppn=15" specifies a limit of just 1 node, and all 15 cores on that node. In fact, on the PSC greenfield machine, the number of cores should always be 15. In fact, your job won't run with any other choice.

After the PBS commands come a sequence of commands that you might imagine typing in interactively; that is, these might be the normal sequence of UNIX commands you would issue to run a particular job.

Briefly, if you have a PBS script file called fred.sh you can submit it to the queue by a command like

        qsub fred.sh

You can check to see the status of your job by the command

        qstat

        qstat -uUSERNAME

When the job is completed, you should find an output file in your directory containing the output, or the error messages that explain why you didn't actually get any output.

Licensing:

The information on this web page is distributed under the MIT license.

Related Data and Programs:

cplex_slurm_arc, examples which uses the slurm() job scheduler to submit a cplex() job to Virginia Tech's Advanced Research Computing (ARC) computer cluster.

mpi, a C code which uses the MPI application program interface for carrying out parallel computations in a distributed memory environment.

openmp, a Fortran90 code which uses the OpenMP application program interface for carrying out parallel computations in a shared memory environment.

slurm_h2p, examples which demonstrate the use of the SLURM batch job scheduler for the h2p computer cluster, as administered by the Center for Research Computing (CRC) at the University of Pittsburgh.

slurm_rcc, examples which use slurm, which is a job scheduler for batch execution of jobs on the FSU Research Computing Center (RCC) computer cluster.

Source Code:

ENVIRON is a batch job script that simply queries the values of certain environment variables, in particular PBS_O_WORKDIR, which can be useful when trying to set up a program to run under the batch system.

environ.sh, the job script.
environ.txt, the output file.

FENICS_BVP_01 is a batch job script that invokes the FENICS program with the input file bvp_01.py, and an auxilliary program timestamp.py

fenics_bvp_01.sh, the job script.
bvp_01.py, the input file to Python FENICS.
fenics_bvp_01.txt, the output file.
bvp_01_mesh.xml, an XML file, output by the script, containing information about the vertices (nodes) and cells (elements) that define the mesh.
bvp_01_u.xml, an XML file, output by the script, containing "function data" (finite element coefficients associated with a mesh), including the index of the data item, the value of the data, the index of the cell (element), and the index of the cell's degree of freedom (internal node).

FENICS_POISSON is a batch job script that invokes the FENICS program with the input file poisson.py, and an auxilliary program timestamp.py.

fenics_poisson.sh, the job script.
poisson.py, the input file to Python.
fenics_poisson.txt, the output file.

FREEFEM_MPI is a batch job script that runs the parallel version of FreeFem++ with the input file schwarz_mpi.edp.

freefem_mpi.sh, the job script.
schwarz_mpi.edp, the input file.
schwarz_mpi_mesh.png, a PNG file of the mesh.
schwarz_mpi_u.png, a PNG file of the solution.
freefem_mpi.txt, the output file.

FREEFEM_SEQUENTIAL is a batch job script that runs the sequential version of FreeFem++ with the input file membrane.edp.

freefem_sequential.sh, the job script.
membrane.edp, the input file.
membrane_mesh.png, a PNG file of the mesh.
membrane_u.png, a PNG file of the solution.
freefem_sequential.txt, the output file.

HELLO is a batch job script that compiles and runs a "hello" program.

hello.sh, the job script.
hello.txt, the output file.

HELLO_MPI illustrates the compilation and execution of a program that includes MPI directives.

hello_mpi.sh, the job script.
hello_mpi.txt, the output file.

HELLO_OPENMP illustrates the compilation and execution of a program that includes OpenMP directives.

hello_openmp.sh, the job script.
hello_openmp.txt, the output file.

MATLAB_PARALLEL runs MATLAB with a parfor command for parallel execution.

matlab_parallel.sh, the job script.
matlab_parallel.m, a MATLAB M file to be executed.
matlab_parallel_log.txt, the log file (MATLAB output).
matlab_parallel.txt, the output file.

POWER_TABLE shows how a MATLAB program can be run through the batch system. We prepare a file of input commands, and invoke MATLAB on the commandline.

power_table.sh, the job script.
power_table.m, a MATLAB M file to be executed.
power_table_log.txt, the log file (MATLAB output).
power_table.txt, the output file.

Last revised on 30 September 2024.