Skip to content

Queue Policies

This page details the QOS and queue usage policies. Examples for each type of job are available.

Jobs are submitted to different queues depending on the queue constraints and the user's desired outcomes. Most jobs are submitted to the "regular" queue, but a user with a particularly urgent scientific emergency may decide to submit to the premium queue for faster turnaround. Another user who does not need the results of this run for many weeks may elect to use the low queue to cut down on costs. And a user who needs fast turnaround while they are using the large telescope could prearrange with NERSC to use the realtime queue for these runs.

These different purposes are served by what is known as "Quality of Service" (QOS): each queue has a different service level in terms of priority, run and submit limits, walltime limits, node-count limits, and cost. The QOS factor is a multiplier that is used in the computation of charges. In exchange for better turnaround time, a user is charged extra to use premium; in exchange for their flexibility in runtime, a user of the flex queue is rewarded with a substantial discount.

Intended Use

There are many different QOS at NERSC, each with a different purpose as outlined below.

Debug

The "debug" QOS is to be used for code development, testing, and debugging. Production runs are not permitted in the debug QOS. User accounts are subject to suspension if they are determined to be using the debug QOS for production computing. In particular, job "chaining" in the debug QOS is not allowed. Chaining is defined as using a batch script to submit another batch script.

Interactive

The "interactive" QOS is to be used for code development, testing, and debugging in an interactive batch session. Jobs should be submitted via salloc -q interactive along with other salloc flags (such as number of nodes, node feature, and walltime request, etc.).

Premium

The intent of the premium QOS is to allow for faster turnaround for unexpected scientific emergencies where results are needed right away. NERSC has a target of keeping premium usage at or below 10 percent of all usage. Premium should be used infrequently and with care. Starting in AY 2021, the charge factor for premium will increase once a project has used 20 percent of its allocation on premium. PIs will be able to control which of their users can use premium for their allocation. Note that premium jobs are not eligible for discounts.

Low

The intent of the "low" QOS is to allow non-urgent jobs to run with a lower usage charge.

Flex

The intent of the “flex" QOS is to encourage user jobs that can produce useful work with a relatively short amount of run time before terminating. For example, jobs that are capable of checkpointing and restarting where they left off may be able to use the flex QOS. Note that this QOS is available only on Cori KNL.

Benefits to using the flex QOS include: The potential to improve your throughput by submitting jobs that can fit into the cracks in the job schedule; A discount in charging for your job.

You can access the flex queue by submitting with -q flex. In addition, you must specify a minimum running time for this job of 2 hours or less with the --time-min flag. Because the walltime you receive may vary, we recommend implementing checkpoint/restart capabilities within your code or using DMTCP to checkpoint your code. Jobs submitted without the --time-min flag will be automatically rejected by the batch system. The maximum wall time request limit (requested via --time or -t flag) for flex jobs must be greater than 2 hours and cannot exceed 48 hours.

Example

A flex job requesting a minimum time of 1.5 hours, and max wall time of 10 hrs:

sbatch -q flex --time-min=01:30:00 --time=10:00:00 my_batch_script.sl

Overrun

The intent of the overrun QOS is to allow users with a zero or negative balance in one of their projects to continue to run jobs. The overrun QOS is not available for jobs submitted against a project with a positive balance. The charging rate for this QOS is 0 and it has the lowest priority on all systems.

If you meet the above criteria, you can access the overrun queue by submitting with -q overrun (-q shared_overrun for the shared queue). In addition, you must specify a minimum running time for this job of 4 hours or less with the --time-min flag. We recommend you implement checkpointing in your overrun jobs to save your progress. Jobs submitted without these flags will be automatically rejected by the batch system.

Example

An overrun job requesting a minimum time of 1.5 hours:

sbatch -q overrun --time-min=01:30:00 my_batch_script.sl

Queues and QOS on Cori

Haswell

QOS Max nodes Max time (hrs) Submit limit Run limit Priority QOS Factor Charge per Node-Hour
regular 1932 48 5000 - 4 1 140
shared 0.5 48 10000 - 4 1 140
interactive1 64 4 2 2 - 1 140
debug 64 0.5 5 2 3 1 140
premium4 1772 48 5 - 2 2 -> 4 280
overrun2 1772 48 5000 - 5 0 0
xfer 1 (login) 48 100 15 - - 0
bigmem 1 (login) 72 100 1 - - 0
realtime3 custom custom custom custom 1 custom custom

KNL

QOS Max nodes Max time (hrs) Submit limit Run limit Priority QOS Factor Charge per Node-Hour
regular 9489 48 5000 - 4 1 80
interactive1 64 4 2 2 - 1 80
debug 512 0.5 5 2 3 1 80
premium4 9489 48 5 - 2 2 -> 4 160[^5]
low 9489 48 5000 - 5 0.5 40[^6]
flex 256 48 5000 - 6 0.25 20
overrun2 9489 48 5000 - 7 0 0

Tip

Jobs in the regular queue using 1024 or more KNL nodes receive a 50% discount!

Note

Jobs in the "shared" QOS are only charged for the fraction of the node used.

Note

User held jobs that were submitted more than 12 weeks ago will be deleted.

JGI Accounts

There are 192 Haswell nodes reserved for the "genepool" and "genepool_shared" QOSes combined. Jobs run with the "genepool" QOS uses these nodes exclusively. Jobs run with the "genepool_shared" QOS can share nodes.

QOS Max nodes Max time (hrs) Submit limit Run limit Priority
genepool 16 72 500 - 3
genepool_shared 0.5 72 500 - 3

Charging

Jobs charges are a function of the number of nodes and the amount of time used by the job, as well as the job's QOS factor and the machine charge factor. For more information on how jobs are charged please see the Computer Usage Charging section of the NERSC usage charging policy.

Warn

For users who are members of multiple NERSC projects, charges are made to the default project, as set in Iris, unless the #SBATCH --account=<NERSC project> flag has been set.

Note

Jobs are charged only for the actual walltime used. That is, if a job uses less time than requested, the corresponding project is charged only for the actual job duration.

Charging discounts

Charging discounts are applied to large KNL jobs, as well as "low", "flex", and "overrun" jobs:

  • The "regular" QOS charges on Cori KNL are discounted by 50% if a job uses 1024 or more nodes.

  • The "low" QOS (available on Cori KNL only) is charged 50% as compared to the "regular" QOS, but no extra large job discount applies.

  • The "flex" QOS (available on Cori KNL only) is charged 25% as compared to the "regular" QOS.

  • The "overrun" QOS is free of charge and is only available to projects that are out of allocation time. Please refer to the overrun section for more details.

Wait times

Queue wait times for past jobs can be a useful guide in estimating wait times of current jobs.


  1. Batch job submission is not enabled and the 64-node limit applies per project not per user. 

  2. The "overrun" QOS is available only when running a job would cause the project (not only the user's allowed fraction) balance to go negative. For overrun jobs a --time-min of 4hrs or less is required. 

  3. The "realtime" QOS is only available via special request - it is only intended for jobs that are connected with an external realtime component that need on-demand processing. 

  4. The charge factor for "premium" QOS is doubled once a project has spent more than 20 percent of its allocation in "premium".