THE LQCD CLUSTER AT UNIVERSITY OF MARYLAND

Department of Physics
University of Maryland
College Park, MD

Cluster Portal

Hardware and Software Overview

The Maryland QCD cluster currently consists of a file-server node, 16 client nodes and a gateway machine.
Each node is a 2.0 GHz dual Pentium 4 Xeon processor server (400 MHz system bus) based on the Super P4DPE motherboard from SuperMicro using Intel's E7500 chipset.
The primary vendor was Build to Order Manufacturing.
Each node has one 20 GB hard disk drive and 0.5 GB of RAM.
The server has a 1.3 TB RAID array mounted as /data, visible from all machines.
On each node, users have access to a 20GB /scratch partition on the local disk.
The nodes are connected via a 24-port Cisco 2950W ethernet switch with each node having an intel PRO/1000 card.
Cluster setup was greatly eased by the use of a Belkin 16-port KVM Omniview PRO2 switch.
The operating system is Red Hat 7.3 Linux (kernel 2.4.18-10smp).
The OpenPBS batch system is used with the Underlord scheduler.
Available compilers:
- Gnu C version 2.6 /usr/bin/gcc
- Gnu C version 3.2 /opt/gcc-3.2/bin
- Gnu C++ version 2.96 /usr/bin/g++
- Gnu C++ version 3.2 /opt/gcc-3.2/bin
- Gnu Fortran77 version 2.96 /usr/bin/g77
- Gnu Fortran77 version 3.2 /opt/gcc-3.2/bin
How to use the QCD Cluster
Logon to the gateway lqcd.phys.umd.edu using secure shell (ssh). Note: This requires a valid account on the University of Maryland Glue System.
The /export/data/lqcd1/ directory on the gateway machine is available for importing files, typically tar-balls, which can be copied to the cluster via scp.
SSH to the file server machine: lqcdfs.
Compile the test programs on lqcdfs. Short jobs (under 15 minutes) may be run on lqcdfs as long as the nice command is used.
The compute nodes should only be used for final production runs and are not for interactive use. Please do not log onto the compute nodes (unless file clean up on local disks is needed due to an error return).
If your executable needs non-standard libraries, you may need to statically link these into your binaries (the -static compiler option.) Commonly-used shared library files are availble on the compute nodes.
Submit jobs to the compute nodes using the qsub command on lqcdfs. The procedure to do this is described in detail below, but it involves writing a shell-script, say run_job, then issuing the command qsub run_job
The UMD cluster currently has one queue.
Jobs can be monitored using the text-based qstat command or the GUI xpbs and xpbsmon.
To kill a job, use the qdel command. When writing your job script, ensure that termination by qdel is handled gracefully.
IMPORTANT: NFS bottlenecks can cause great problems with the PBS queue. Please DO NOT use cp to transfer files between the server and the compute nodes. Instead, use scp and specify lqcdfs: in the server file name.

Writing the job shell-script file (serial jobs)

See the qsub man page for details on composing the job script.
For serial jobs, the script file should perform the following sequence of tasks:

Issue the necessary PBS directives.
Stage in: copy all needed files, including the executable , into the directory /scratch/PBS_<job_id> which PBS automatically creates on the local disk on the compute node.
Run the program executable (in the foreground, not in the background).
Stage out: after program execution is done, copy appropraite files from the local disk back into the user's permanent directory; the /scratch/PBS_<job_id> directory on the local disk is automatically deleted by PBS upon job completion.
Provide some clean up instructions in case the job must be killed using qdel.

See the sample PBS script for more details. A downloadable version of this file is provided which you can edit to suit your needs (right-click for menu to download and save).

Further information about PBS

Job management is performed using the Portable Bath System (PBS).
Manual pages for PBS commands:
qsub | qstat | qdel | qalter | qhold | qmsg | qmove | qrls | qrerun | qselect | qsig
Details about PBS to assist you in writing your job scripts may be found in the PBS User's Guide
PBS External Reference Specification
PBS Administrator Guide
PBSPro Administrator Guide
PBS Internal Design Specification

THE LQCD CLUSTER AT UNIVERSITY OF MARYLAND

Department of Physics University of Maryland College Park, MD

Hardware and Software Overview

How to use the QCD Cluster

Writing the job shell-script file (serial jobs)

Further information about PBS

The GNU Compilers

Department of Physics
University of Maryland
College Park, MD