The U of A UITS group maintains High Performance and High Throughput Computing systems for use in research applications. A shared memory supercomputer and two distinct Linux clusters are available for running jobs requiring a large amount of memory, parallel processing, and certain visualization and scientific applications.
Serial jobs (1 processor) and threaded jobs using up to 12 threads and 24Gb of memory will run successfully on a single node of the HTC (High Throughput Computing) system. Programs that use the MPI (Message Passing Interface) library can take advantage of multiple nodes on the HPC Cluster, using up to 24 Gb memory per node. Non-MPI codes that need more than 24Gb of memory can run on the HPC Shared Memory system. Details about the hardware and accessing the supercomputer resources are listed on the HPC web page. A chart showing allowable CPU and memory requests for jobs is HERE.
Some modules are different, for example, to use BioPerl on the new systems, load the perl module (there is no separate bioperl module.) The PBS job scheduler is more flexible, and if one of the HPC/HTC systems has a high load, PBS may run a job on a different system that has a smaller job load.
Accounts are available to UA faculty and faculty-sponsored individuals. A UA NetID is required. It may be possible for outside collaborators to get "Designated Campus Colleagues" (DCC) Accounts. See the DCC info page and the DSV info page.
To get a UA HPC/HTC account, you need a UANetID and a faculty sponsor. The accounts page is: https://account.arizona.edu/. A single account gives you will automatically have access to the HTC, ICE cluster, and Shared Memory systems. If you are a faculty member or PI and wish to sponsor an account, the link is: http://sponsor.hpc.arizona.edu/
From a Linux or MacOSX system, you can login from a Terminal window by typing the command: ssh -X myUANetID@login.hpc.arizona.edu (On Mac, go to Applications and search for Terminal.)
From a Windows computer, first download the XMing application from http://sourceforge.net/projects/xming/ and then download the PuTTy application from https://sitelicense.arizona.edu/nocost.php Run XMing to allow display windows to open on your computer, e.g. as needed by graphical applications such as the nedit editor. Open PuTTy, and enter login.hpc.arizona.edu for the Hostname. In the box a the left, under Connection. open the SSH option and select X11. Click the 'Enable X11 forwarding' checkbox. At left, scroll back up to the top and select Session. In the Saved Sessions box, type hpc and click the Save button. When you open PuTTy form this point on, you can select the Saved session (hpc) and click Open (your X11 settings will be retained when you Save a named Session.) Enter your UANetID and password (no characters will echo while you are typing the password.)
Once you see a prompt ending with a 'greater than' sign, you're logged in. To logout, type 'logout' (and be sure to logout when finished if you are using a publicly accessible computer.
The shell on HPC systems is taken from the choice of shell
on the U-system account. To change the shell on HPC, one can log
in to u.arizona.edu (ssh YourUANetID@u.arizona.edu) and type
and be guided to select one of the relevant shells. It may take
up to an hour to propagate to the other systems. Logout from u.arizona.edu with the command 'logout'
The default disk quota is 5Gb. You can monitor your disk usage by running the uquota utility. You can set up larger temporary allocations using the xdisk utility. See the HPC/HTC Storage page for more information on xdisk. To receive notification that your temporary xdisk quota is about to expire, create a file named .forward in your home directory on the supercomputer and put your email address in the .forward file. It is also possible to lease or purchase storage from UITS.
The nedit editor is very easy to use. It does require X11 forwarding (see above question about login.) You can run the jpico editor as an alternative, but most people find nedit easier to use.
Jobs are placed in a PBS (Portable Batch System) batch queue by the 'qsub' command. Jobs remain queued until the requested resources are available. When a job finishes, an email notification is sent to the email address specified in the PBS submit script.
Applications and utilities installed on the supercomputer include Perl/BioPerl, the EMBOSS suite, the NCBI BLAST suite of applications, blat, fasta, HMMER, phylip, clustalw, mp34compsw (Smith-Waterman search), mrbayes-MPI. Run the commands 'module load unsupported' then 'module avail' to see which applications have been installed as modules.
Contact HPC-Consult AT list.arizona.edu to find out if a particular application can be installed on the supercomputer.
Use sftp, WinSCP, CyberDuck or other file transfer program to transfer your data files to the supercomputer (See below). To log into the supercomputer use the ssh command: ssh -X myUANetID@login.hpc.arizona.edu or use PuTTy (See above.)
Jobs are submitted to the Portable Batch System (PBS) via script files. Some example scripts can be found HERE. There is a sample BLAST submission script in /genome/ICEblast.csh. A sample mrbayesMPI submit file is in /genome/mrbayesMPI.csh. Use the cp command to copy one of these sample files to your directory and edit your copy as follows. Use the va (view allocations) command to identify your group_list setting for the .csh file. Edit your .csh file to specify the JobName, email address, group_list, cpu and time requests (See next question), and commands to run. To submit your job, type the command: ‘qsub script’ (do not type the quotation marks). The qsub command displays the job number assigned to your submission.
These are directives to the PBS batch system, and need to be in the script. The lines must begin with ‘#PBS’, and you must alter the jobname (following -N option) and your email address (following the –M option).The –l options identify requested resources and the –q option requests a particular job queue.
The –m bea directive tells the batch system to send email notifications when your job begins, ends, or aborts. The –W line identifies your work group, which can be found using the va (view allocations) command.
Use the command ‘qstat -a’ to view all jobs, or 'qstat -u myUANetID' to view your jobs. For more details on resource usage by a particular job, run 'qstat -f jobNumber'
The command ‘qdel job_number’ will stop your job and remove it from the queue.
The batch system creates an output file and an error file, named by appending the letter o or e and the job number to the jobname specified in the PBS script. The output file contains the standard output from the job, possibly including the line ‘Warning: no access to tty (Bad file number).’ This warning can be ignored. The error file will contain more specific information about the error that occurred. It also may include the line ‘stty: tcgetattr: Not a typewriter’, which can be ignored.
‘/pbs/mom1/mom_priv/jobs/1754...SC: Command not found.’ ?
This type of error can occur if you have edited your script or data files on a PC and transferred them to a Unix system. Check for control-M characters (^M) at the end of lines in your script by using the command: ‘cat –vet script’.
This command reads input from the file ‘PCscript’ and writes output to the file ‘UNIXscript’. Do not try to use the same filename for both input and output or your input file will be corrupted.
One possiblilty is that your group has already used the 10000 cpu hours allocated for the default queue. The va command will show you the amount of time remaining. If you have insufficient time remaining in the default queue, you can try submitting your job to the windfall queue.
From a Mac, use CyberDuck or Fugu. From Windows, use WinSCP. WinSCP can be downloaded from http://sitelicense.arizona.edu
See the FAQ Running BLAST jobs on the Supercomputer.
There are many useful links at the Research Computing web site.
Email us at bcf AT arl.arizona.edu or the HPC Help Desk at HPC-Consult AT list.arizona.edu. Include the exact error messages generated by your job and the locations of your script and data files.
Please email us at bcf AT arl.arizona.edu to report any answers that are not clear or any errors that may be present in this document.