NERSC User Environment¶
Home Directories, Shells and Dotfiles¶
All NERSC systems use global home directories, which are are pre-populated with shell initialization files (also known as dotfiles) for all available shells. NERSC supports
tcsh as login shells. Other shells (
zsh) are also available. The default shell at NERSC is bash.
The standard dotfiles are symbolic links to read-only files that NERSC controls. For each standard dotfile, there is a user-writeable ".ext" file.
~/.cshrc are read-only. Customizations should be put in
.ext scheme used at NERSC is scheduled for change in February 2020, see below.
Occasionally, a user will accidentally delete the symbolic links to the standard dotfiles, or otherwise damage the dotfiles to the point that it becomes difficult to do anything. In this case, the user should run the command
/usr/common/software/bin/fixdots. This will recreate the original dotfile configuration, after first saving the current configuration in the directory
$HOME/KeepDots.timestamp is a string that includes the current date and time. After running
fixdots, the user should carefully incorporate the saved customizations into the newly-created .ext files.
Dotfile changes planned in February 2020¶
To reduce the shell startup overhead, and to bring NERSC in line with other HPC centers, NERSC will NOT reserve your home dotfiles for system use anymore starting from February 2020 (The exact transition date is still TBA). The NERSC settings in the current home dotifles will be migrated to the
/etc/profile.d directory on each system. After this transition, you can use your home
.bashrc, etc., for your personal shell modifications (You don't have to work with those
.ext files anymore!).
Notes for existing user accounts¶
To mitigate the interruptions to your workloads, we will do the following to preserve your current shell environment on the transition day.
We will replace your dotfiles, that are currenly symbolic links to the files under NERSC control with template dotfiles that source their .ext files. For example, here is how your ~/.bashrc file would look like,
# begin .bashrc if [[ -z "$SHIFTER_RUNTIME" ]] then . $HOME/.bashrc.ext fi # end .bashrc
You are recommended to move what's in your ~/.bashrc.ext file into your .bashrc file after the transition (and remove the .ext files after the move). If you do not use shifter applications, then you can remove the if block for testing shifter runtime.
If you have already been using local dotfiles (i.e., not using the dotfiles reserved by the system), we will not make any changes to your dotifles. You may find the altd and darshan modules are loaded in your environment, but you can unload them in your dotfiles if you prefer.
While we expect that this transition is transparent to most users, some users may experience changes in their shell environment. We have configured cori12 (one of the Cori login nodes) and dtn12 (DTN test node), so that you can test your shell environment now before the actual changes take place. You can type the following commands to get the dotfiles that will be populated on your account on the transition day (type dotmgr -h for more info):
dotmgr -s # save your current dotfiles, and print the location dotmgr -e # get the dotfiles that will be populated on your account on the transition day
You can then login to cori12 and/or dtn12 to check whether this affected your environment.
You can then return to your current dotfiles after testing with the following command:
dotmgr -r <directory-that-the-save-step-returned>
You'll need to log out of cori12/dtn12, and log back into a Cori/DTN login node, to get your restored dotfiles take effect.
Notice that now you can test your new dotfiles on the login nodes only. Since the environment on the login nodes are passed to the compute nodes by the batch system, you will see similar shell environment on the compute nodes. We will arrange compute nodes access in next January for you to test as well. Please let us know of any problems you encounter, by filing a ticket at help.nersc.gov
Notes for new user accounts created after February 2020¶
If your NERSC account is created after the dotfile migration (February 2020), NERSC will not populate any dotfiles on your home directory. You need to create your own dotfiles (e.g., .bashrc, .bash_profile, etc) as needed to put your personal shell modifications.
NERSC loads darshan (a lightweight I/O profiling tool) and altd (a library tracking tool) by default. If you encounter any problems with them, you can unload them in your .bash_proifle, or .login file:
module unload darshan module unload altd
If you run shifter applications, you many want to skip the dotfiles on your home. You can use the following if block in your dotfiles (for bash)
if [[ -z "$SHIFTER_RUNTIME" ]]; then #put your Cori setting here ... fi
If any of the NERSC defined environment variables, e.g., SCRATCH, are missing in your shell invocations, you can add them in your ~/.bashrc file as follows:
if [ -n "$SCRATCH" ]; then export SCRATCH=/global/cscratch1/sd/$USER fi
Changing Default Login Shell¶
Use NIM to change your default login shell. Login, then select Change Shell from the Actions pull-down menu.
NERSC Modules Environment¶
NERSC uses the module utility to manage nearly all software. There are two huge advantages of the module approach:
- NERSC can provide many different versions and/or installations of a single software package on a given machine, including a default version as well as several older and newer version.
- Users can easily switch to different versions or installations without having to explicitly specify different paths. With modules, the
MANPATHand related environment variables are automatically managed.
The following is a list of commands available in the Modules Environment tool available on Cori.
To get a usage list of module options type the following (listing is abbreviated):
module help Available Commands and Usage: + add|unload modulefile [modulefile …] + rm|unload modulefile [modulefile …] + switch modulefile1 modulefile2 + display modulefile [modulefile …] + avail path [path] + list + help modulefile [modulefile …]
# To get all available packages module avail # To know the availability of a specific software module avail netcdf # To know all packages that contain a substring, use -S flag. module avail -S netcdf
To see what changes are made to your environment when a module is loaded:
module display [modulefile]
module show [modulefile]
This command will add one or more modulefiles to your current environment. It does so silently, but will throw errors if there are any problems with the modulefile. If you load the generic name of a module, you will get the default version. To load a specific version, load the modulefile using the full specification.
module load [modulefile1][modulefile2] # Load visit module load visit # Load visit version 2.1.2 module load visit/2.1.2
Unloads the specified modulefile from the user's environment. This command will fail silently if the modulefile you specify is not already loaded.
module unload [modulefile]
The modules environment allows you to swap between versions of packages
module swap [old modulefile] [new modulefile]
Creating a Custom Environment¶
You can modify your environment so that certain modules are loaded whenever you log in. Put changes in one of the following files, depending on your shell:
Users may have certain customizations that are appropriate for one NERSC platform, but not for others. This can be accomplished by testing the value of the environment variable
$NERSC_HOST. For example, on Cori the default programming environment is Intel (PrgEnv-Intel). A C-shell user who wants to use the
GNU programming environment should include the following module command in their
if ($NERSC_HOST == "cori") then module swap PrgEnv-intel PrgEnv-gnu endif
Install Your Own Customized Modules¶
You can create and install your own modules for your convenience or for sharing software among collaborators. The module definition files can be placed in the following locations:
- project directory
- your home directory
- available file system.
Make sure the UNIX file permissions grant access to all users who want to use the software.
Do not give write permissions to your home directory to anyone else.
As an example, we have modulefile named myzlib located in
To register this modulefile with our modules environment we run the following commands:
nersc$ module use /global/project/projectdirs/mpccc/usg/modulefiles/cori nersc$ module load myzlib/1.2.7
module use command adds this new directory before other module search paths (defined as
$MODULEPATH), so modules defined in this custom directory will have precedence if there are other modules with the same name in the module search paths. If you prefer to have the new directory added at the end of
module use -a instead of