File System Quotas and Purging¶

NERSC sets quotas on file systems shown in the table below. Purged file system are purged of files that have not been accessed in the time period shown below.

Overview¶

Policy¶

NERSC data management policy.

Quotas¶

You may briefly exceed your quota on Perlmutter scratch by as much as 10 TB for up to 24 hours before no new data can be written. This is known as a 'grace period'. When you are above your 20 TB quota, showquota --grace will show how much time you have left in your grace period before the file system becomes read-only.

Quota tools¶

NERSC offers two separate tools to retrieve your quota status:

• showquota, which works on Cori, Perlmutter and Data Transfer Nodes, offers more control over the file systems to retrieve quota for, works with all file systems general users have access to (including the HPSS archive!) and provides many features, such as overquota grace times and JSON output.

• myquota and the specialized tools cfsquota/prjquota/cmnquota were designed for Cori. On Perlmutter and DTNs those commands still appear as available, but they are actually symlinks to showquota, which has been instructed to alter its default output format depending on the name of the script it's called from, so users should notice almost no difference.
  The hpssquota command is the only one that changed output content and format with the inclusion into showquota, compared to the version on Cori.
  Cori users can continue to use the old myquota, cfsquota and cmnquota executables on Cori or use the symlinks to showquota on the other systems, but are invited to transition to showquota to benefit of the more advanced features and prepare for Perlmutter.

To see current usage for the Home and Scratch file systems (Cori Scratch on Cori an DTNs, Perlmutter Scratch on Perlmutter) you can just type:

showquota

You can also exclude the Home and Scratch file systems from the default file systems to report quotas for with -N/--no-default-fs.

For the Community File System you can specify the projects you want to check quota for with:

showquota PROJECT [PROJECT [..]]

For the Common File System you can add --cmn, and showquota will report quotas for all projects you pass as arguments:

showquota --cmn PROJECT [PROJECT [..]]

The --hpss/--hpssquota option makes showquota report HPSS Archive quotas for all projects the user is part of:

$ showquota --hpss
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                                  File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+
| username usage on HPSS charged to project123 |   23.74TiB |     1.44PiB |           1.6% |          - |           - |              - |
| username usage on HPSS charged to project456 |      0.00B |   133.00TiB |           0.0% |          - |           - |              - |
+----------------------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Or alternatively you can specify full paths, and mix and match different quota queries into a single one (unfortunately there's no way to request HPSS quotas by path, only by enabling the --hpss keyword):

$ showquota --no-default-fs --full-path $CFS/dasrepo /global/common/software/dasrepo
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|                     File system | Space used | Space quota | Space used (%) | Inode used | Inode quota | Inode used (%) |
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+
|       /global/cfs/cdirs/dasrepo |   98.54TiB |   110.00TiB |          89.6% |     32.51M |     100.00M |          32.5% |
| /global/common/software/dasrepo |   72.99GiB |   100.00GiB |          73.0% |    819.56K |       2.00M |          41.0% |
+---------------------------------+------------+-------------+----------------+------------+-------------+----------------+

Different output formats and unit of measurements are available, see showquota --help.

For scripting purposes we suggest to use the JSON formatting option -J/--json, which can be quite powerful when paired with e.g. the jq utility:

$ showquota -J dasrepo nstaff |jq '.[] | "\(.fs) \(.space_perc)"'
"home 0.2%"
"pscratch 0.7%"
"dasrepo 0.9%"
"nstaff 0.4%"

The -L/--limit/--grace option shows how much more storage and inodes you can temporarily write above your quota and how much time you have left before no new data can be written.

showquota is a Python script, and has been designed with composability in mind, so you can use it in your own Python scripts by making it available in your $PYTHONPATH variable, e.g.:

$ PYTHONPATH="$(dirname $(which showquota)):$PYTHONPATH" python3
>>> import showquota
>>> stats = showquota.report('project123', full_path=True)
>>> stats
[/global/cfs/cdirs/project123: 70.1% storage - 87.9% inode]
>>> stats[0].is_overquota()
False

Feel free to reach out by opening a ticket for more details.

Increases¶

If you or your project needs additional space for your scratch file system or HPSS you may request it via the Disk Quota Increase Form.

Quotas on the Community File System are determined by DOE Program Managers based on information PIs supply in their yearly ERCAP requests. If you need a mid-year quota increase on the Community File System, please use the Disk Quota Increase Form link above and we will pass the information along to the appropriate DOE Program Manager for approval.

Purging¶

Some NERSC file systems are purged. This means the files not read (i.e. atime is updated) within a certain time period are automatically deleted. You can see the time period for the purged file systems at NERSC in the overview table. When a purge is done, a file named .purged.<date> is left behind. This is a text file that holds the names of the files that have been removed. These .purged files will not be deleted by our purges to make sure a record of purging activities is retained. Touching files or other actions intended to circumvent the purge are forbidden by NERSC policy.

If you want to see the purged files from $SCRATCH you can do the following, it will report a timestamp of when the purge was performed. Note files from $SCRATCH can't be retrieved since there is no backup, however you can see what files were purged if you cat the content of file.

elvis@cori> ls $SCRATCH/.purged*
/global/cscratch1/sd/elvis/.purged.20210125  /global/cscratch1/sd/elvis/.purged.20210822