Skip to content

Queue Policies

This page details the QOS and queue usage policies. Examples for each type of job are available.

Jobs are submitted to different queues depending on the queue constraints and the user's desired outcomes. Most jobs are submitted to the "regular" queue, but a user with a particularly urgent deadline may decide to submit to the premium queue for faster turnaround. Another user who does not need the results of this run for many weeks may elect to use the low queue to cut down on costs. And a user who needs fast turnaround while they are using the large telescope could prearrange with NERSC to use the realtime queue for these runs.

These different purposes are served by what is known as "Quality of Service" (QOS): each queue has a different service level in terms of priority, run and submit limits, walltime limits, node-count limits, and cost. The QOS factor is a multiplier that is used in the computation of charges. In exchange for better turnaround time, a user is charged extra to use premium; in exchange for their flexibility in runtime, a user of the flex queue is rewarded with a substantial discount.

Intended Use

There are many different QOS at NERSC, each with a different purpose as outlined below.

Debug

The "debug" QOS is to be used for code development, testing, and debugging. Production runs are not permitted in the debug QOS. User accounts are subject to suspension if they are determined to be using the debug QOS for production computing. In particular, job "chaining" in the debug QOS is not allowed. Chaining is defined as using a batch script to submit another batch script.

Interactive

The "interactive" QOS is to be used for code development, testing, and debugging in an interactive batch session. Jobs should be submitted via salloc -q interactive along with other salloc flags (such as number of nodes, node feature, and walltime request, etc.).

Premium

The intent of the "premium" QOS is to allow for faster turnaround before conferences and urgent project deadlines. It should be used with care. NERSC has a target of keeping premium usage at or below 10 percent of all usage. Note that premium jobs are not eligible for discounts.

Low

The intent of the "low" QOS is to allow non-urgent jobs to run with a lower usage charge.

Flex

The intent of the “flex" QOS is to encourage user jobs that can produce useful work with a relatively short amount of run time before terminating. For example, jobs that are capable of checkpointing and restarting where they left off may be able to use the flex QOS. Note that this QOS is available only on Cori KNL.

Benefits to using the flex QOS include: The potential to improve your throughput by submitting jobs that can fit into the cracks in the job schedule; A discount in charging for your job.

You can access the flex queue by submitting with -q flex. In addition, you must specify a minimum running time for this job of 2 hours or less with the --time-min flag. Because the walltime you receive may vary, we recommend implementing checkpoint/restart capabilities within your code or using DMTCP to checkpoint your code. Jobs submitted without the --time-min flag will be automatically rejected by the batch system. The maximum wall time request limit (requested via --time or -t flag) for flex jobs must be greater than 2 hours and cannot exceed 48 hours.

Example

A flex job requesting a minimum time of 1.5 hours, and max wall time of 10 hrs:

sbatch -q flex --time-min=01:30:00 --time=10:00:00 my_batch_script.sl

Overrun

The intent of the overrun QOS is to allow users with a zero or negative balance in one of their repositories to continue to run jobs. The overrun QOS is not available for jobs submitted against a repository with a positive balance. The charging rate for this QOS is 0 and it has the lowest priority on all systems.

If you meet the above criteria, you can access the overrun queue by submitting with -q overrun (-q shared_overrun for the shared queue). In addition, you must specify a minimum running time for this job of 4 hours or less with the --time-min flag. We recommend you implement checkpointing in your overrun jobs to save your progress. Jobs submitted without these flags will be automatically rejected by the batch system.

Example

An overrun job requesting a minimum time of 1.5 hours:

sbatch -q overrun --time-min=01:30:00 my_batch_script.sl

Queues and QOS on Cori

Haswell

QOS Max nodes Max time (hrs) Submit limit Run limit Priority QOS Factor Charge per Node-Hour
regular 1932 48 5000 - 4 1 140
shared1 0.5 48 10000 - 4 1 140
interactive4 64 4 2 2 - 1 140
debug 64 0.5 5 2 3 1 140
premium 1772 48 5 - 2 2 280
overrun2 1772 48 5000 - 5 0 0
xfer 1 (login) 48 100 15 - - 0
bigmem 1 (login) 72 100 1 - - 0
realtime3 custom custom custom custom 1 custom custom

KNL

QOS Max nodes Max time (hrs) Submit limit Run limit Priority QOS Factor Charge per Node-Hour
regular 9489 48 5000 - 4 1 805
interactive4 64 4 2 2 - 1 80
debug 512 0.5 5 2 3 1 80
premium 9489 48 5 - 2 2 1605
low 9489 48 5000 - 5 0.5 406
flex 256 48 5000 - 6 0.25 20
overrun2 9489 48 5000 - 7 0 0

Tip

Jobs in the regular queue using 1024 or more KNL nodes receive a 50% discount!

Note

User held jobs that were submitted more than 12 weeks ago will be deleted.

JGI Accounts

There are 192 Haswell nodes reserved for the "genepool" and "genepool_shared" QOSes combined. Jobs run with the "genepool" QOS uses these nodes exclusively. Jobs run with the "genepool_shared" QOS can share nodes.

QOS Max nodes Max time (hrs) Submit limit Run limit Priority
genepool 16 72 500 - 3
genepool_shared 0.5 72 500 - 3

Charging

Jobs charges are a function of the number of nodes and the amount of time used by the job, as well as the job's QOS factor and the machine charge factor. For more information on how jobs are charged please see the Computer Usage Charging section of the NERSC usage charging policy.

Warn

For users who are members of multiple NERSC repositories charges are made to the default account, as set in Iris, unless the #SBATCH --account=<NERSC repository> flag has been set.

Example

A job which ran for 35 minutes on 3 KNL nodes on Cori with the regular QOS would be charged: $$ (35/60)\ \text{hours}*3\ \text{nodes} * 80 = 140\ \text{NERSC hours} $$

Example

A job which ran for 12 hours on 4 physical cores (each core has 2 hyperthreads) on Cori Haswell with the shared QOS would be charged: $$ 12\ \text{hours} * (2*4\ \text{cores}/64) * 140 = 210\ \text{NERSC hours} $$

Note

Jobs are charged only for the actual walltime used. That is, if a job uses less time than requested, the corresponding account is charged only for the actual job duration.


  1. Jobs in the "shared" QOS are only charged for the fraction of the node used. 

  2. The "overrun" QOS is available only when running a job would cause the repository (not only the user's allowed fraction) balance to go negative. For overrun jobs a --time-min of 4hrs or less is required. 

  3. The "realtime" QOS is only available via special request - it is only intended for jobs that are connected with an external realtime component that need on-demand processing. 

  4. Batch job submission is not enabled and the 64-node limit applies per repository not per user. 

  5. The "regular" QOS charges on Cori KNL are discounted by 50% if the job uses 1024 or more nodes. 

  6. The "low" QOS (available on Cori KNL only) is charged 50% as compared to the "regular" QOS, but no extra large job discount applies.