• For the (new) arrow cluster, add the following line to .bashrc (and make sure to use the bash shell):

• arrow cluster: 55 machines, two 3.2 GHz CPUs each, 2GB memory, running Linux.
  

• ssh onto a submit host (arrowhead.cs.ubc.ca). Note that the tech staff has restricted ssh access to some hosts of the department (for security reasons). arrowhead is only available locally. To SSH to arrowNN, you must first SSH to arrowhead.

   qsub -cwd -S /bin/bash -q all.q -P eh -o ./ ./helloworld.sh

   qstat -u "*"
   

qstat -u "*"
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 359476 0.56000 helloworld seramage     r     05/07/2014 17:59:30 all.q@arrow06.cs.ubc.ca            1        

If the state is r, then the job is running. If you see many jobs it may be held in state qr.

• <--*Like it or not, you can't submit ridiculously long jobs*: Please keep unrestricted jobs in the range of a few minutes or maximally around an hour. Once a job runs on the cluster it will run until completion (except jobs longer than 25 hours which are automatically killed); this means that if you submit unrestricted long jobs you will completely block the cluster for all other users. So please try to split jobs up into array jobs (see the instructions on array jobs) where each single task takes just a few minutes. If you have to submit longer jobs, please use a consumable, such as max20 (see below).-->

  As of Feb 2014, the arrow cluster is not as heavily contended as it once was. The usage is normally coordinated by gentlemanly agreement based on group priorities. Arrow will not try to kill your long jobs as well.

Multi-core jobs (Arrow Cluster)

• For the (new) arrow cluster, add the line

     source /cs/sungridengine/arrow/common/settings.csh
     

  to your configuration file (e.g. ~/csh_init/cshrc), or add

1. klb: jobs of Kevin Leyton-Brown's students.

Manually limiting the number of your jobs (Arrow Cluster)

Manually limiting the number of your jobs (ICICS/BETA Clusters)

NEW

1. kpmUrgent: urgent jobs of Kevin Murphy's students. If you use this priority class, please use the kpmUrgent consumable (i.e. add -l kpmUrgent=1 to your SGE command; see 'priority classes' below); if you don't you will block the cluster for everyone else.

NEW

• [-h] place user hold on job
• [-hold_jid job_identifier_list] define jobnet interdependencies

The [-h] option will hold your array of jobs in the queue until you have NO other jobs running on the SGE. The [-hold_jid job_identifier_list] will hold your array of jobs in the queue until a SPECIFIC job(s) is finished, defined by the job_identifier_list. So, to manually limit the number of jobs you are running (lets say 10 jobs) but NOT limit yourself to the max10 consumable, split your array up to blocks of 10 and submit them with the [-h] option. This will ensure that you never use more than 10 machines at a time.

• ssh onto a submit host. For beta, submit hosts are (we're not sure), for arrow it is samos (which is now another name for arrow). Note that the tech staff has restricted ssh access to some hosts of the department (for security reasons), and the samos and arrowNN are affected by that. SSH access to arrowNN is restricted to connections coming from samos (note that you shouldn't actually be connecting to anything other than arrow01, and even that machine may have some restrictions), and SSH access to samos is restricted to connections from begbie and okanagan.

• If you want your script to include some more fancy commands rather than just call your program, you have to add the line #$ -S /bin/sh to your script, or you have to add -S /bin/sh as an argument to qsub. For example, if your script has a while, it won't work unless your script has the following two lines at the beginning:

     #!/bin/sh
     #$ -S /bin/sh
     

  or it has only the first line #!/bin/sh, and you call it like this:

     qsub -cwd -S /bin/sh script.sh
     

• Similarly, you can submit a Perl script instead of a shell script (Yeah, I like that too!), you just have to start your script by

     #!/cs/local/bin/perl
     #$ -S /cs/local/bin/perl
     

  or you can add -S /cs/local/bin/perl to your qsub command and remove the second line in the script (the first one still needs to be there).

• Probably it would work with other scripting languages too!

I hate shell scripts! Can I use Perl instead? Or some other scripting language?

• Luckily, YES! Read the previous section "How do I submit jobs".

• Like it or not, you can't submit ridiculously long jobs: Please keep unrestricted jobs in the range of a few minutes or maximally around an hour. Once a job runs on the cluster it will run until completion (except jobs longer than 25 hours which are automatically killed); this means that if you submit unrestricted long jobs you will completely block the cluster for all other users. So please try to split jobs up into array jobs (see the instructions on array jobs) where each single task takes just a few minutes. If you have to submit longer jobs, please use a consumable, such as max20 (see below).

Common Problems

• Your job has the status "Eqw"
  • Try _dos2unix_ing the file that your queue commands are in

• If you just want to see if the cluster works for you, use the following command. When the job finishes, an output file will be written to the current directory.

• If you don't want to store the output and error files for your job (be cautious though!), then you can throw them away directly, i.e. send them to /dev/null:

     qsub -cwd -o /dev/null -e /dev/null script.sh
     

• If you want to submit to a specific queue, for example to have same-power machines, specify -q queue_name. The name of icics is icics.q, and the name of beta is beta.q:

     qsub -cwd -o /dev/null -e /dev/null -q icics.q script.sh
     

CVS

How to set up CVS.

Memory intensive jobs

Manually limiting the number of your jobs

What is the configuration of the machines, and how busy are they?

http://samos.cs.ubc.ca/ganglia/

1. Urgent: intended for very occasional use, mostly around paper deadlines. The cluster is limited to 10 such jobs at any time. Please use the urgent consumable (see 'priority classes' below).

1. kpm: jobs of Kevin Murphy's students.
2. kpmUrgent: urgent jobs of Kevin Murphy's students. Please use the kpmUrgent consumable (see 'priority classes' below).

   qsub -cwd -o <outfiledir> -e <errorfiledir> -P kpmUrgent -l kpmUrgent=1 helloworld.sh
   

Arrow users: CFI Blurbs

The Arrow cluster was funded under a CFI grant. This grant requires us to complete annual reports explaining how this infrastructure is being used. If you use the cluster for a project, large or small, please enter a bullet item here that gives a short description of your project and the role the cluster played.

• Sample project: description (students/faculty involved)

   qsub -cwd -o . -e . ~kevinlb/World/helloWorld.sh
   

How do I submit jobs?

What is an appropriate job size?

• Not ridiculously short jobs though!: jobs that are extremely short (eg 0.1 second) might take longer to be dispatched than they take to run, again hurting performance if a gigantic number of such jobs are in the queue. An ideal job length is on the order of a minute (i.e., if your jobs are extremely short, it's best to group them to achieve this sort of length). This would mean that higher-priority jobs would be able to access the cluster within about a second on average, but that job dispatching would not overwhelm the submit host.
• Like it or not, you can't submit ridiculously long jobs: On the arrow cluster, jobs that run longer than 25 hours are automatically killed, but rather try to keep jobs to minutes or hours. This makes it easiest for SGE to help share resources in a fair way. See the instructions on array jobs for how to split up a really long job.

Array jobs

Priority Classes

If you use the qsub syntax above on arrow, your job will be assigned to the default priority class associated with your user account. This cannot be the priority class Urgent. To use another priority class than your default for a job, use the following syntax (note the capital P):

   qsub -cwd -o <outfiledir> -e <errorfiledir> -P <priorityclass> helloworld.sh
   

   qsub -cwd -o <outfiledir> -e <errorfiledir> -P Urgent -l urgent=1 helloworld.sh
   

CPLEX and MATLAB license management

If your job uses CPLEX or Matlab, add -l cplex=1 or -l matlab=1 to your qsub command. This will ensure that we don't run more jobs than there are licenses. Instructions on how to use CPLEX are found here. We're not exactly sure of the best way to run MATLAB without invoking X-windows, but the best we've been able to do is

   matlab -nojvm -nodisplay -nosplash < inputfile.m
   

Right now the cluster allows 22 simultaneous CPLEX processes and (separately) 10 simultaneous Matlab processes. Using the -l syntax above ensures that no more than this number of processes run. If you don't use the syntax, you run the risk of invoking more processes than there are licenses, causing some of your jobs to fail and thus requiring you to rerun jobs. The department as a whole has 100 CPLEX licenses, of which many are in use at any time; thus, running a Matlab job without the -l syntax runs a high chance of failure. We have 22 CPLEX licenses, but they should only be used on this cluster; thus again you'll need to use -l but you shouldn't have to worry about users elsewhere in the dept. To check the number of Matlab licenses available you can type:

   /cs/local/generic/lib/pkg/matlab-7.2/etc/lmstat -a
   

• CPLEX and MATLAB if your job uses CPLEX or Matlab, add -l cplex=1 or -l matlab=1 to your qsub command. This will ensure that we don't run more jobs than there are licenses. Instructions on how to use CPLEX are found here. We're not exactly sure of the best way to run MATLAB without invoking X-windows, but the best we've been able to do is

     matlab -nojvm -nodisplay -nosplash < inputfile.m
     

Introduction

Available clusters

The Arrow cluster

How to submit jobs

How to monitor, control, and delete jobs

Administration

For details on how to administer the cluster (requires admin access), look at SunGridEngineAdmin.

Where do Arrow jobs run?

Right here:

• CPLEX and MATLAB if your job uses CPLEX or Matlab, add -l cplex=1 or -l matlab=1 to your qsub command. This will ensure that we don't run more jobs than there are licenses. Instructions on how to use CPLEX are found here.

1. low: jobs of particularly low priority. This priority class can be used to submit a huge amount of jobs that will give way to any other jobs in the queue if there are any.

1. eh (for empirical hardness): jobs which pertain to the particular project mentioned in the CFI grant under which the cluster was funded.
2. ea (for empirical algorithmics): studies on the empirical properties of algorithms (i.e., "the 'E' in BETA"), but not part of the project described above.

• Priority Class: If you use the qsub syntax above on arrow, your job will be assigned to the default priority class associated with your user account. This cannot be the priority class Urgent. To use another priority class than your default for a job, use the following syntax (note the capital P):

Introduction:

Available clusters:

• beta cluster: 5 machines, two 2GHz CPUs each, 4GB memory, running Linux. Available for members of the beta lab.
  

• icics cluster: 13 machines, two 3GHz CPUs each, 2GB memory, running Linux. Available for all members of the department.
  

• arrow cluster: 50 machines, two 3.2 GHz CPUs each, 2GB memory, running Linux.
  This cluster is tied to Kevin Leyton-Brown's CFI grant for research on empirical hardness models, but is also available to other users in the department when it is idle.

The Arrow cluster:

Jobs running on the arrow cluster belong to one of four priority classes. Jobs are scheduled (selected to be run) pre-emptively by priority class, and then evenly among users within a priority class. (Note that scheduling among users is done on the basis of CPU usage, not on the basis of the number of jobs submitted. Thus a user who submits many fast jobs will be scheduled more often than a user at the same priority class who submits many slow jobs.) Because of the preemptive scheduling, users submitting to a lower priority class may see high latency (it may take hours or days before a queued job is scheduled). On the other hand, these lower priority jobs will be allocated all 100 CPUs when no higher-priority jobs are waiting. All users should feel free to submit as many jobs as they like, as doing so will not interfere with the cluster's ability to serve its primary purpose.

The priority classes are:

1. Urgent: intended for very occasional use, mostly around paper deadlines. The cluster is limited to 10 such jobs at any time.
2. EmpiricalHardness: jobs which pertain to the particular project mentioned in the CFI grant under which the cluster was funded.
3. EmpiricalAlgorithmics: studies on the empirical properties of algorithms (i.e., "the 'E' in BETA"), but not part of the project described above.
4. general: jobs which do not fall into one of the above categories. We ask that these jobs be relatively short in duration (although there can be arbitrarily many of them): since new jobs can only be scheduled when a processor becomes idle, excessively long low-priority jobs can lead to starvation of high-priority jobs.

In order to submit in any priority class (even 'general'), access for that class must be explicitly granted to your user account. To request access, please contact Frank Hutter, Lin Xu or Kevin Leyton-Brown.

How to submit jobs:

• ssh onto a submit host. For beta, submit hosts are (we're not sure), for arrow it is samos (which is now another name for arrow)

How to monitor, control, and delete jobs:

• A job is submitted to the cluster in the form of a shell (.sh) script.
  
• You can either submit single or array jobs. An example for a single job would e.g. be the one-line script

• Array jobs are useful to submit many similar jobs. In general, you should prefer array jobs over multiple single jobs.
  An example of an array job is the one-line script

     echo 'Hello world, number ' $SGE_TASK_ID
     

  If this is the content of the file many-helloworld.sh, you can submit an array job by typing

     qsub -cwd -o <outfiledir> -e <errorfiledir> -t 1-100 many-helloworld.sh
     

  on the command line, where the range 1-100 is chosen arbitrarily here. This will create a new array job with an automatically assigned job number <jobnumber> and 100 entries that is queued. Each entry of the array job will eventually run on a machine in the cluster - the <i>th entry will be called <jobnumber>.<i>. Sungrid Engine treats every entry of an array job as a single job, and when the <i>th entry is called assigns <i> to the variable $SGE_TASK_ID. You may use this variable to do arbitrarily complex things in your shell script - an easy option is to index a file and execute the <i>th line with the <i>th job.

How to monitor, control, and delete jobs:

• The command qstat is used to check the status of the queue. It lists all running and pending jobs. There is an entry for each entry of a running array job, whereas pending parts of array jobs are listed in one line. qstat -f gives detailed information for each cluster node, qstat -ext more detailed information for each job. Try man qstat for more options.
  
• The command qmon can be used to get a graphical interface to monitor and control jobs. It's not great, though.
• The command qdel can be used to delete (your own) jobs. (syntax: qdel <jobnumber>). You can also delete single entries of array jobs (syntax qdel <jobnumber>.<i>).

Urgent jobs on arrow If you have an urgent deadline and would like to use some CPUs with comparably low latency, please contact Kevin Leyton-Brown, Frank Hutter, or Lin Xu to be temporarily added as an urgent user. Once you are added as a temporary urgent user, you can submit jobs using qsub -P Urgent -l urgent=1 instead of qsub. As said above, the total number of urgent jobs is limited to 10. This is even true if those are the only jobs on the cluster - in that case (which probably will never happen because many users will have stuff waiting) you can still fill the rest of the cluster with "normal" jobs.

-- FrankHutter and Lin Xu - 02 May 2006

SunGridEngine - quick user guide

Introduction:

This page gives a quick overview of the available computational facilities and how to use them with the SunGridEngine scheduling software.

General policies:

• Submit short jobs.
  SGE is not a load balancing software. Once a job is running it runs with 100% of a CPU until it's done.
  Thus, if you submit many long jobs, you will block the cluster for other users. Do not do that.
  Due to the share-based scheduling we've set up, your overall share of computational time will not be larger if you submit larger jobs.
  On the arrow cluster, jobs that run longer than 25 hours are automatically killed, but rather try to keep jobs to minutes or hours. This makes it easiest for SGE to help share resources in a fair way.

Available clusters:

• beta cluster: 5 machines, two 2GHz CPUs each, 4GB memory, running Linux. Available for members of the beta lab.
  This should probably be used only via SGE (with a share-based scheduling system that will actually work, as opposed to the current first-come-first-serve scheme)
  
• icics cluster: 13 machines, two 3GHz CPUs each, 2GB memory, running Linux. Available for all members of the department.
  Some people run stuff locally on these machines. We could still use SGE on top of that (it dispatches jobs based on load), but there is no guarantee to get 100% CPU time on the nodes you're running on.
  
• arrow cluster: 50 machines, two 3.2 GHz CPUs each, 2GB memory, running Linux.
  This cluster is tied to Kevin Leyton-Brown's CFI grant for research on empirical hardness models. Because of this, jobs of this kind are required to have higher priority than other jobs. However, when no such jobs are running there are 100 CPUs up for grabs.
  This cluster is thus appropriate to run many (small) jobs that do not require a quick turn-around. When submitting to this cluster you have to expect to wait for weeks until your jobs run - but then you probably get the whole cluster for a couple of days.
  This cluster is not appropriate for jobs that require low latency. For special requests, such as very near paper deadlines, we set up a temporary urgent passing lane jobs in which will be run first. To prevent urgent jobs from blocking the whole cluster, their total number is limited to 10.

Details about the machines, their configuration, and their names: Ganglia

How to submit jobs:

• For the arrow cluster, add the line

     source /cs/beta/lib/pkg/sge-6.0u7_1/default/common/settings.csh
     

  to your configuration file (e.g. ~/csh_init/.cshrc). For the beta cluster, the appropriate line to add is

     source /cs/beta/lib/pkg/sge/beta_grid/common/settings.csh
     

  but we may completely get rid of this configuration and have it all in one. Currently, you work solely with the one cluster that is indicated by this line in the configuration file - there is no easy way to go back and forth (but this will hopefully change).
• ssh onto a submit host. For beta, submit hosts are ?, for arrow it is samos (which is now another name for arrow)
• A job is submitted to the cluster in the form of a shell (.sh) script A simple script could e.g. be

     echo 'Hello world.'
     

  If this is the content of the file helloworld.sh, you submit it by typing

     qsub -cwd -o <outfiledir> -e <errorfiledir> helloworld.sh
     

  on the command line, where is the directory the job's output file (in this case containing "Hello world") is written to, and is the directory any error output is written to. The outfile will have the name helloworld.sh.o, the errorfile the name helloworld.sh.e (where the job number is automatically assigned by SGE).