Skip to content

E4S 22.05 - Perlmutter

The e4s-22.05 stack is built using the Spack branch e4s-22.05 on Perlmutter using GCC and CCE compilers. This stack can be loaded by running:

module load e4s/22.05

You can also load the same software stack via module load spack/e4s-22.05

Note

Going forward, we do not generate modules for Spack. In order to use E4S software you must use spack load to load a package in your user environment.

Here is a breakdown of the installed specs by spack environments:

Spack Environments Compiler Root Specs Implicit Specs Total Specs
gcc gcc@11.2.0, gcc@10.3.0 55 262 317
cuda gcc@11.2.0 17 97 114
nvhpc nvhpc@22.5 6 19 25
cce cce@15.0.0 8 64 72

Spack Configuration

When you load the e4s/22.05 module you will see several spack environments which will enable access to one of the sub-stacks. To see a list of spack environments you can run the following:

elvis@perlmutter> spack env list
==> 4 environments
    cce  cuda  gcc  nvhpc

You can run spack env status to retrieve your active spack environment. The message below indicates you are not in any active spack environment therefore you can't access the software stack.

elvis@perlmutter> spack env status
==> No active environment

For reference, please see Environments Tutorial for further assistance.

For instance, if you want to access the nvhpc spack environment you can do the following.

elvis@perlmutter> spack env activate -V nvhpc
elvis@perlmutter> spack env status
==> In environment nvhpc

The Spack configuration is accessible on the filesystem by changing directories into the root of the spack environment. For example, if you want to find the spack configuration for the nvhpc environment, you can run the following:

elvis@perlmutter> spack cd -e nvhpc
elvis@perlmutter> pwd && ls -l
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/var/spack/environments/nvhpc
total 56
-rw-rw-r-- 1 e4s spackecp 48395 Jan 11 13:51 spack.lock
-rw-rw-r-- 1 e4s spackecp  8172 Jan 11 13:50 spack.yaml

The file spack.yaml contains the configuration used to build all the spack environments. You can find all of our spack configurations in the Spack Infrastructure project.

Overview

You can see a list of all available packages via spack find. Shown below is breakdown of packages by all spack environments

Preview of the gcc spack environment
elvis@perlmutter> spack -e gcc find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%gcc@11.2.0 +blosc+bzip2~cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release /wlbzh44
amrex@22.05%gcc@11.2.0 ~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials~tiny_profile build_type=RelWithDebInfo dimensions=3 precision=double /ofkinx3
butterflypack@2.1.1%gcc@11.2.0 ~ipo+shared build_type=RelWithDebInfo /4kznqwp
ccache@4.5.1%gcc@11.2.0 ~ipo build_type=RelWithDebInfo /ybl7xef
cdo@2.0.5%gcc@11.2.0 ~curl~external-grib1+fftw3+hdf5+libxml2~magics+netcdf+openmp+proj+szip+udunits2 grib2=eccodes /mifq7z4
conduit@0.8.3%gcc@11.2.0 ~adios+blt_find_mpi~doc~doxygen+fortran+hdf5+hdf5_compat~ipo+mpi+parmetis~python+shared~silo+test~zfp build_type=RelWithDebInfo /wazineh
dyninst@12.1.0%gcc@11.2.0 ~ipo+openmp~stat_dysect~static build_type=RelWithDebInfo /imsyy3c
fortrilinos@2.0.0%gcc@11.2.0 +hl~ipo+shared build_type=RelWithDebInfo /gr4hjrx
gasnet@2022.3.0%gcc@11.2.0 ~cuda~debug~rocm conduits=smp /bwmz3br
gawk@5.1.1%gcc@11.2.0 ~nls /44kt3vx
git@2.35.3%gcc@11.2.0 +man+nls+perl+subtree~svn~tcltk /ubv32cm
gmake@4.3%gcc@11.2.0 ~guile+nls /btwzv5s
gnuplot@5.4.3%gcc@11.2.0 +X+cairo+gd+libcerf~pbm~qt~wx patches=ad89f23 /bsmjqym
grads@2.2.1%gcc@11.2.0 +geotiff+shapefile /lbkrgcs
gromacs@2021.5%gcc@11.2.0 ~blas~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~plumed~relaxed_double_precision+shared~sycl build_type=RelWithDebInfo /oi4vmzo
gsl@2.7%gcc@11.2.0 ~external-cblas /fhx3zdz
hdf5@1.10.7%gcc@11.2.0 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /6opvtfc
heffte@2.2.0%gcc@11.2.0 ~cuda+fftw~fortran~ipo~magma~mkl~python~rocm+shared build_type=RelWithDebInfo /atk3gky
hpctoolkit@2022.04.15%gcc@11.2.0 ~all-static~cray~cuda~debug~level_zero~mpi+papi~rocm+viewer /aijvids
hpx@1.7.1%gcc@11.2.0 ~async_cuda~async_mpi~cuda~examples~generic_coroutines~ipo~rocm~tools build_type=RelWithDebInfo cxxstd=17 instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=mpi /yqlat2t
hypre@2.24.0%gcc@11.2.0 ~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory /szss2c5
kokkos@3.6.00%gcc@11.2.0 ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper build_type=RelWithDebInfo std=14 /3b4inm2
kokkos-kernels@3.6.00%gcc@11.2.0 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /ss2tvzi
lammps@20220107%gcc@11.2.0 ~asphere~body~class2~colloid~compress~coreshell~cuda~cuda_mps~dipole~exceptions+ffmpeg~granular~ipo+jpeg+kim~kokkos~kspace~latte+lib~manybody~mc~meam~misc~mliap~molecule+mpi~mpiio~opencl+openmp~opt~peri+png~poems~python~qeq~replica~rigid~shock~snap~spin~srd~user-adios~user-atc~user-awpmd~user-bocs~user-cgsdk~user-colvars~user-diffraction~user-dpd~user-drude~user-eff~user-fep~user-h5md~user-lb~user-manifold~user-meamc~user-mesodpd~user-mesont~user-mgpt~user-misc~user-mofff~user-netcdf~user-omp~user-phonon~user-plumed~user-ptm~user-qtb~user-reaction~user-reaxc~user-sdpd~user-smd~user-smtbq~user-sph~user-tally~user-uef~user-yaff~voronoi build_type=RelWithDebInfo /h34ece6
libquo@1.3.1%gcc@11.2.0  /jdmo2xh
metis@5.1.0%nvhpc@22.5 ~gdb~int64~real64+shared build_type=Release patches=4991da9 /qxtdo74
nano@4.9%gcc@11.2.0  /3ajsqyz
nccmp@1.9.0.1%gcc@11.2.0 ~ipo build_type=RelWithDebInfo /svlmona
ncl@6.6.2%gcc@11.2.0 ~gdal~hdf4+openmp+triangle+udunits2 patches=64f3502,a2f7ac8,a612d41 /zmspv5c
nco@5.0.1%gcc@11.2.0 ~doc /t4avmjn
ncview@2.1.8%gcc@11.2.0  /5ujvhbd
nwchem@7.0.2%gcc@11.2.0 ~mpipr~openmp /hhxzzid
openblas@0.3.20%gcc@11.2.0 ~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=openmp /lrsjsel
openpmd-api@0.14.4%gcc@11.2.0 ~adios1+adios2+hdf5~ipo+mpi~python+shared build_type=RelWithDebInfo /q4ob2r7
parallel@20210922%gcc@11.2.0  /arb2fpm
parmetis@4.0.3%gcc@11.2.0 ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f89253,50ed208,704b84f /jdqvpwn
pdt@3.25.1%gcc@11.2.0 ~pic /4wzbhsa
petsc@3.17.1%gcc@11.2.0 ~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C /z7wbk6n
plasma@21.8.29%gcc@11.2.0 ~ipo~lua+shared build_type=RelWithDebInfo /sht5ri2
py-libensemble@0.9.1%gcc@10.3.0 ~deap~mpi~mpmath~nlopt~petsc4py~pyyaml~scipy~tasmanian /xy4w2mx
py-warpx@22.05%gcc@11.2.0 +mpi /nteuogz
py-warpx@22.05%gcc@11.2.0 +mpi /rmebedc
py-warpx@22.05%gcc@11.2.0 +mpi /atxm4s3
qthreads@1.16%gcc@11.2.0 +hwloc~spawn_cache+static scheduler=distrib stack_size=4096 /xsyqv4g
quantum-espresso@7.0%gcc@11.2.0 +cmake~elpa~environ+epw~gipaw~ipo~libxc+mpi~openmp+patch~qmcpack+scalapack build_type=RelWithDebInfo hdf5=none /olvwavb
raja@0.14.0%gcc@11.2.0 ~cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo /idylhfo
slate@2021.05.02%gcc@11.2.0 ~cuda~ipo+mpi+openmp~rocm+shared build_type=RelWithDebInfo /eh6rahv
slepc@3.17.1%gcc@11.2.0 +arpack~blopex~cuda~rocm /uetioli
subversion@1.14.1%gcc@11.2.0 +nls~perl~serf /fg7keqv
sundials@6.2.0%gcc@11.2.0 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /kfepsit
superlu@5.3.0%gcc@11.2.0 ~ipo+pic build_type=RelWithDebInfo /ziz42oa
tasmanian@7.7%gcc@11.2.0 ~blas~cuda~fortran~ipo~magma~mpi+openmp~python~rocm~xsdkflags build_type=Release /4yefyvt
upcxx@2022.3.0%gcc@11.2.0 ~cuda~gasnet~mpi~rocm cross=none /yjru4k5
vtk-m@1.7.1%gcc@11.2.0 ~64bitids+ascent_types~cuda+doubleprecision~ipo~kokkos~logging~mpi+openmp+rendering~rocm~shared~tbb~testlib~virtuals build_type=Release /zvu6y2s
xterm@353%gcc@11.2.0  /s6hwjyj
Preview of the nvhpc spack environment
elvis@perlmutter> spack -e nvhpc find --format "{name}@{version}%{compiler} {variants} /{hash:7}"
hdf5@1.10.7%nvhpc@22.5 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /q7pqnux
kokkos@3.6.00%nvhpc@22.5 ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper build_type=RelWithDebInfo std=14 /lbfztq6
kokkos-kernels@3.6.00%nvhpc@22.5 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /qspw732
sundials@6.2.0%nvhpc@22.5 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /4bb7vwm
superlu@5.3.0%nvhpc@22.5 ~ipo+pic build_type=RelWithDebInfo /gs4k3my
zfp@0.5.5%nvhpc@22.5 ~aligned~c~cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo /a3kwh2u
Preview of the cce spack environment
elvis@perlmutter> spack -e cce find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%cce@15.0.0 +blosc+bzip2~cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release /75i5mf3
hdf5@1.10.7%cce@15.0.0 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /2hv5uoq
hypre@2.24.0%cce@15.0.0 ~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory /ngi2ghh
kokkos-kernels@3.6.00%cce@15.0.0 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /aclkqoy
petsc@3.17.1%cce@15.0.0 ~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C /f5232qk
sundials@6.2.0%cce@15.0.0 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /vigxlll
superlu@5.3.0%cce@15.0.0 ~ipo+pic build_type=RelWithDebInfo /x7icfwr
superlu-dist@7.2.0%cce@15.0.0 ~cuda~int64~ipo+openmp~rocm+shared build_type=RelWithDebInfo patches=8da9e21 /5kwghlm
Preview of the cuda spack environment
elvis@perlmutter> spack -e cuda find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%gcc@11.2.0 +blosc+bzip2+cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release cuda_arch=80 /df2vmcb
hpctoolkit@2022.04.15%gcc@11.2.0 ~all-static~cray+cuda~debug~level_zero+mpi~papi~rocm+viewer /ntu5gi2
hypre@2.24.0%gcc@11.2.0 ~complex+cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory cuda_arch=80 /bonsnuz
kokkos@3.6.00%gcc@11.2.0 ~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning+wrapper build_type=RelWithDebInfo cuda_arch=80 std=14 /lonrnyk
kokkos-kernels@3.6.00%gcc@11.2.0 ~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl~openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=80 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /swui7am
lammps@20220107%gcc@11.2.0 ~asphere~body~class2~colloid~compress~coreshell+cuda+cuda_mps~dipole~exceptions+ffmpeg~granular~ipo+jpeg+kim~kokkos~kspace~latte+lib~manybody~mc~meam~misc~mliap~molecule+mpi~mpiio~opencl+openmp~opt~peri+png~poems~python~qeq~replica~rigid~shock~snap~spin~srd~user-adios~user-atc~user-awpmd~user-bocs~user-cgsdk~user-colvars~user-diffraction~user-dpd~user-drude~user-eff~user-fep~user-h5md~user-lb~user-manifold~user-meamc~user-mesodpd~user-mesont~user-mgpt~user-misc~user-mofff~user-netcdf~user-omp~user-phonon~user-plumed~user-ptm~user-qtb~user-reaction~user-reaxc~user-sdpd~user-smd~user-smtbq~user-sph~user-tally~user-uef~user-yaff~voronoi build_type=RelWithDebInfo cuda_arch=80 /uqdra2u
likwid@5.2.1%gcc@11.2.0 +cuda+fortran /wst4i5t
papi@6.0.0.1%gcc@11.2.0 +cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools /zy7rjq4
parmetis@4.0.3%gcc@11.2.0 ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f89253,50ed208,704b84f /jdqvpwn
pdt@3.25.1%gcc@11.2.0 ~pic /4wzbhsa
petsc@3.17.1%gcc@11.2.0 ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C cuda_arch=80 /nb2ei2k
raja@0.14.0%gcc@11.2.0 +cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo cuda_arch=80 /gueomxk
slate@2021.05.02%gcc@11.2.0 +cuda~ipo+mpi+openmp~rocm+shared build_type=RelWithDebInfo cuda_arch=80 /bdcxawb
slepc@3.17.1%gcc@11.2.0 +arpack~blopex+cuda~rocm cuda_arch=80 /lmzfg45
strumpack@6.3.1%gcc@11.2.0 ~butterflypack+c_interface~count_flops+cuda~ipo+mpi+openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp build_type=RelWithDebInfo cuda_arch=80 /pocpro4
tau@2.31.1%gcc@11.2.0 ~adios2+binutils~comm~craycnl+cuda+elf+fortran~gasnet+io~level_zero+libdwarf+libunwind~likwid+mpi~ompt~opari~opencl~openmp+otf2+papi+pdt~phase~ppc64le~profileparam+pthreads~python~rocm~rocprofiler~roctracer~scorep~shmem~sqlite~x86_64 /5uqetmp
zfp@0.5.5%gcc@11.2.0 ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80 /k4cqxwj

Shown below are a few useful spack find commands you can use to filter output.

All root specs built with the NVHPC compiler
elvis@perlmutter> spack find -x --format "{name}" %nvhpc
hdf5  kokkos  kokkos-kernels  sundials  superlu  zfp

Recommendation on User Environment setup with spack environments

This stack provides the following spack environments: gcc, nvhpc, cce, cuda. The cuda environment provides packages built with CUDA support with gcc@11.2.0 compiler. Depending on the spack environment you are using, please consider loading the appropriate modules

  1. Load PrgEnv-gnu, cpu module when using gcc spack environment
  2. Load PrgEnv-cce, cpu module when using cce spack environment
  3. Load PrgEnv-nvhpc, cpu module when using nvhpc spack environment
  4. Load PrgEnv-gnu, gpu module when using cuda spack environment

When in doubt, please refer to compiler settings to determine the list of modules used. To retrieve a list of compilers you can do spack compiler list. To query a compiler instance like gcc@11.2.0 you can run spack compiler info gcc@11.2.0.

In the cuda spack environment, you will see packages are built with variants +cuda cuda_arch=80 that are optimized for NVIDIA A100 GPUs. You should load craype-accel-nvidia modulefile, which is typically loaded at startup or via gpu modulefile. You can check the accelerator target by running the following:

elvis@perlmutter> echo $CRAY_ACCEL_TARGET
nvidia80

Loading E4S Packages in Your User Environment

We do not generate modules for Spack packages, therefore one must use spack load to load the appropriate package in your user environment. Please refer to the help message (spack load --help) for a list of options.

To see a list of Spack packages loaded in your user environment type spack load --list. The first time you module load e4s no packages will be loaded.

elvis@perlmutter> spack load --list
==> 0 loaded packages

To load a package type spack load and the name of the package. The command spack load works similarly to module load in the sense that it will dynamically change your user environment to access software.

Use Case Example: Loading HDF5

First, let's activate the gcc environment by running the following:

spack env activate -V gcc

Let's assume you want to use HDF5, first you need to determine which Spack package you want to load. We can run the following to show the installed versions of HDF5.

spack find hdf5

This will output a list of installed packages as shown:

==> 5 installed packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
hdf5@1.8.22  hdf5@1.10.7  hdf5@1.10.8  hdf5@1.12.2  hdf5@1.12.2

Let's load HDF5 version 1.10.7 that was compiled with GCC version 11.2.0 into our user environment. To specify the version use @ and to specify the compiler use %. These symbols can also be combined as is the case with loading GCC version 11.2.0. Take note that Spack will load the dependencies in your user environment which is the default behavior.

elvis@perlmutter> spack load hdf5@1.10.7%gcc@11.2.0
elvis@perlmutter> spack load --list
==> 20 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6   cray-mpich@8.1.22  expat@2.4.8  libarchive@3.5.2  libiconv@1.16  libuv@1.44.1  lzo@2.10        ncurses@6.1    rhash@1.4.2  zlib@1.2.12
cmake@3.23.1  curl@7.66.0        hdf5@1.10.7  libbsd@0.11.5     libmd@1.0.4    lz4@1.9.3     mbedtls@2.28.0  pkgconf@1.8.0  xz@5.2.5     zstd@1.5.2

Your user environment will be updated. That includes PATH, LD_LIBRARY_PATH, and other variables relevant for loading the package. The HDF5 package provides the h5cc program which will appear in your environment as shown below.

elvis@perlmutter> which h5cc
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/h5cc
dir is /global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne
gcc (GCC) 11.2.0 20210728 (Cray Inc.)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

This confirms that HDF5 was successfully loaded.

View Shell Commands to Load the Package

It can be informative to view the commands that Spack uses when a package is loaded. For example, to see the shell commands used when zlib is loaded, use spack load --sh zlib%gcc@11.2.0.

elvis@perlmutter> spack load --sh zlib%gcc@11.2.0
export CMAKE_PREFIX_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/cmake-3.23.1-p23fzuowp4yuitemelic7f65nwybthxd:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/mbedtls-2.28.0-ucarxpp47pj7yo7abt5povg3swygev3q:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Systems/2022.2.1:/global/common/software/nersc/pm-2021q4/easybuild/software/Nsight-Compute/2022.1.1;
export LD_LIBRARY_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/lib:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/mbedtls-2.28.0-ucarxpp47pj7yo7abt5povg3swygev3q/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/lib:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/lib:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/math_libs/11.7/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/extras/CUPTI/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/extras/Debugger/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/nvvm/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/lib64:/global/common/software/nersc/pm-2022q3/sw/darshan/3.4.0/lib:/opt/cray/pe/papi/6.0.0.16/lib64:/opt/cray/pe/gcc/11.2.0/snos/lib64:/opt/cray/libfabric/1.15.2.0/lib64;
export MANPATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/share/man:/usr/share/man:/usr/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libiconv-1.16-3xwztgyaaco4thqqhacdst65mmnr5is3/share/man:/usr/share/man:/usr/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/share/man:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/share/man:/usr/share/man:/usr/man:/opt/nvidia/hpc_sdk/Linux_x86_64/22.5/cuda/11.7/doc/man:/opt/cray/pe/perftools/22.09.0/man:/opt/cray/pe/papi/6.0.0.16/share/pdoc/man:/opt/cray/pe/gcc/11.2.0/snos/share/man:/opt/cray/pe/craype/2.7.19/man:/opt/cray/pe/mpich/8.1.22/ofi/man:/opt/cray/pe/mpich/8.1.22/man/mpich:/opt/cray/pe/libsci/22.11.1.2/man:/opt/cray/pe/man/csmlversion:/opt/cray/pe/dsmml/0.2.2/dsmml/man:/opt/cray/libfabric/1.15.2.0/share/man:/usr/share/lmod/lmod/share/man:/usr/local/man:/usr/share/man:/usr/man:;
export PKG_CONFIG_PATH=/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/hdf5-1.10.7-6opvtfcoaxox7oskb7dzisnb32fnk5ne/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q/lib/pkgconfig:/opt/cray/pe/mpich/8.1.22/ofi/gnu/9.1/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/rhash-1.4.2-xjxbsefahiue4isqlzwcobgc7sgfhmqm/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libuv-1.44.1-gitz4v6soymrlpymzvygqlqibktfsp7a/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libarchive-3.5.2-d4gpdx2rxj4calconu7cd6kgq2ev5roe/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zstd-1.5.2-rtaozj7xdfhxengzppbqbth4mya3edpq/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/zlib-1.2.12-ozmcyfjfv7i5gjjgklfsh43h67vzsuc5/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/xz-5.2.5-tyswlp4epqidsxgtgxyefciqoz2ju6oe/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lzo-2.10-rlpwurcqberlfjlpk62go3arnneonv6u/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/lz4-1.9.3-xxbkaphnofa6gxugxszqr5kzew7chrqa/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/expat-2.4.8-yvchkv23jw7po7meyuxckpre5uroipmd/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libbsd-0.11.5-uomjj7io3hhggbfyzofrw5jpw5chlv74/lib/pkgconfig:/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/libmd-1.0.4-iynqjjkz4jgtxagqy2o2tvspqfn3yvog/lib/pkgconfig:/usr/share/pkgconfig:/usr/lib64/pkgconfig:/usr/lib/pkgconfig:/opt/modulefiles/cudatoolkit:/global/common/software/nersc/pm-2022q3/sw/darshan/3.4.0/lib/pkgconfig:/opt/cray/pe/craype/2.7.19/pkg-config:/opt/cray/pe/dsmml/0.2.2/dsmml/lib/pkgconfig:/opt/cray/libfabric/1.15.2.0/lib64/pkgconfig;
export SPACK_LOADED_HASHES=ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:6opvtfcoaxox7oskb7dzisnb32fnk5ne:nugfov2wsvpllczfy6ff7bytzt4oi34q:6yvclv72nfvhlx2fj72n5npigrgouojd:p23fzuowp4yuitemelic7f65nwybthxd:xjxbsefahiue4isqlzwcobgc7sgfhmqm:i6ri5efvxr243dakip6bk7n6b3qfruyb:gitz4v6soymrlpymzvygqlqibktfsp7a:d4gpdx2rxj4calconu7cd6kgq2ev5roe:rtaozj7xdfhxengzppbqbth4mya3edpq:ozmcyfjfv7i5gjjgklfsh43h67vzsuc5:tyswlp4epqidsxgtgxyefciqoz2ju6oe:ucarxpp47pj7yo7abt5povg3swygev3q:rlpwurcqberlfjlpk62go3arnneonv6u:xxbkaphnofa6gxugxszqr5kzew7chrqa:3xwztgyaaco4thqqhacdst65mmnr5is3:4g7s6qpmt2nsclyi2d2ndfek2ptey4m6:yvchkv23jw7po7meyuxckpre5uroipmd:uomjj7io3hhggbfyzofrw5jpw5chlv74:iynqjjkz4jgtxagqy2o2tvspqfn3yvog:zkrv7nh3jglxsimtfzld7vo4mhdqlh3u;

Find the Install Prefix for a Package

For your application you may want to specify the location of the installed package as an environment variable. In this case, let's set HDF5_DIR to the parent directory of libs and include --the install prefix for our HDF5 package. A convenient way to do this is as follows,

elvis@perlmutter> export HDF5_DIR=$(spack location -i hdf5@1.10.7%gcc@11.2.0)

Unloading Spack Packages From Your Environment

You can use spack unload to remove packages from your environment which functions similarly to module unload. spack unload will only work on installed specs that were loaded in your environment. In the example below, we show how to unload the cmake Spack package from the user environment. Take note as we unload cmake via spack unload cmake the user environment was updated and it now resorts to the system cmake /usr/bin/cmake

elvis@perlmutter> spack load --list
==> 20 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6   cray-mpich@8.1.22  expat@2.4.8  libarchive@3.5.2  libiconv@1.16  libuv@1.44.1  lzo@2.10        ncurses@6.1    rhash@1.4.2  zlib@1.2.12
cmake@3.23.1  curl@7.66.0        hdf5@1.10.7  libbsd@0.11.5     libmd@1.0.4    lz4@1.9.3     mbedtls@2.28.0  pkgconf@1.8.0  xz@5.2.5     zstd@1.5.2

elvis@perlmutter> which cmake
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/opt/spack/cray-sles15-zen3/gcc-11.2.0/cmake-3.23.1-p23fzuowp4yuitemelic7f65nwybthxd/bin/cmake

elvis@perlmutter> spack unload xz@5.2.5

elvis@perlmutter>  which cmake
/usr/bin/cmake

elvis@perlmutter> spack load --list
==> 19 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6  cray-mpich@8.1.22  curl@7.66.0  expat@2.4.8  hdf5@1.10.7  libarchive@3.5.2  libbsd@0.11.5  libiconv@1.16  libmd@1.0.4  libuv@1.44.1  lz4@1.9.3  lzo@2.10  mbedtls@2.28.0  ncurses@6.1  pkgconf@1.8.0  rhash@1.4.2  xz@5.2.5  zlib@1.2.12  zstd@1.5.2

If you want to remove all packages from your user environment you can run the following:

elvis@perlmutter> spack unload --all
elvis@perlmutter> spack load --list
==> 0 loaded packages

Using Hashes for Packages with Multiple Versions

Sometimes, you will need to use hashes instead of specs to load a specific package variant. Let's assume we want to load CMake and we get the error, "Use a more specific spec". Note that in this example we have three specs for same version but each has a different hash (The 7 character code proceeding the spec).

Error

elvis@perlmutter> spack load cmake@3.23.1%gcc@11.2.0
==> Error: cmake@3.23.1%gcc@11.2.0 matches multiple packages.
  Matching packages:
    p23fzuo cmake@3.23.1%gcc@11.2.0 arch=cray-sles15-zen3
    6v4d2hu cmake@3.23.1%gcc@11.2.0 arch=cray-sles15-zen3
  Use a more specific spec.

In this case, we can use the hash to uniquely identify the desired variant of CMake. For example, if we want to load the spec with hash p23fzuo, we would run the following:

elvis@perlmutter> spack load /p23fzuo

Loading a Spack Package Without Dependencies

The command spack load will load a package and its dependencies which is recommended when using software, however sometimes you don't want to load all the extra dependencies in your environment. Spack provides a way to load only the package via --only package which will load the package and not its dependencies. In the example shown below, we load one package. Take note of this compared to its normal behavior which loads all dependencies.

elvis@perlmutter> spack load --list
==> 0 loaded packages
elvis@perlmutter> spack load --only package /p23fzuo
elvis@perlmutter> spack load --list
==> 1 loaded package
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
cmake@3.23.1

elvis@perlmutter> spack unload -a
elvis@perlmutter> spack load --list
==> 0 loaded packages
elvis@perlmutter> spack load /p23fzuo
elvis@perlmutter> spack load --list
==> 17 loaded packages
-- cray-sles15-zen3 / gcc@11.2.0 --------------------------------
bzip2@1.0.6  cmake@3.23.1  curl@7.66.0  expat@2.4.8  libarchive@3.5.2  libbsd@0.11.5  libiconv@1.16  libmd@1.0.4  libuv@1.44.1  lz4@1.9.3  lzo@2.10  mbedtls@2.28.0  ncurses@6.1  rhash@1.4.2  xz@5.2.5  zlib@1.2.12  zstd@1.5.2

Spack Views

Spack environments can define filesystem views to directly access the software without having to load software.

If you are in an existing spack environment, you will want to deactivate and reactivate spack environment with view. In this example, let's enable spack view for nvhpc spack environment

!!! note: Activating spack environment with view may take some time depending on size of spack environment

elvis@perlmutter> despacktivate
elvis@perlmutter> spack env activate --with-view nvhpc

Spack will automatically load everything in the view, Take note, we have 0 loaded packages in the output of spack load --list, however we can access software from the view since $PATH, $LD_LIBRARY_PATH, etc... has been updated in your user environment. Spack will present software in a view similar to how one access software in Linux in /usr/local or /usr/bin.

elvis@perlmutter> spack load --list
==> 0 loaded packages

elvis@perlmutter> which h5cc
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/var/spack/environments/nvhpc/views/default/hdf5/1.10.7-nvhpc-22.5/bin/h5cc

elvis@perlmutter> which cmake
/global/common/software/spackecp/perlmutter/e4s-22.05/75197/spack/views/default/cmake/3.23.1-gcc-11.2.0/bin/cmake

Now you may be wondering which packages you can access with spack views. We have selected all gcc specs in the view with some exceptions, for instance we can't load two instances of same package such as cmake in the same view or two packages built with gcc and cce compiler or two packages one built with CUDA and without.

Developer Workflow

Shown below is a guide to get you started building packages with our Spack instance. First, we recommend you source the following file which will activate a python environment and install clingo in your environment.

# bash, sh, zsh users
source $SPACK_ROOT/bin/spack-setup.sh

# csh or tcsh users
source $SPACK_ROOT/bin/spack-setup.csh

Upon sourcing the file you will see the following message, note that clingo is installed as a Python package which can be seen by running pip list.

elvis@perlmutter> source $SPACK_ROOT/bin/spack-setup.sh
Creating python virtual environment in /global/homes/s/elvis/.spack-pyenv
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/s/elvis/.spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
Package       Version
------------- -------
cffi          1.15.1
clingo        5.5.2
mpi4py        3.1.2
numpy         1.22.3
periodictable 1.5.0
picmistandard 0.0.19
pip           20.2.3
pycparser     2.21
pyelftools    0.26
pyparsing     3.0.6
scipy         1.8.0
setuptools    59.4.0
toml          0.10.2
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/global/homes/s/elvis/.spack-pyenv/bin/python3 -m pip install --upgrade pip' command.
Your python interpreter used by spack is /global/homes/s/elvis/.spack-pyenv/bin/python

We recommend you run the following to see if Spack will concretize a spec and not try to bootstrap clingo. Note we must exit the e4s environment via despacktivate otherwise Spack will use the production path which will cause permission issues. You can confirm your active environment by running spack env status. Next try running spack spec zlib to check if spack will concretize zlib which will use clingo installed in your Python environment.

(.spack-pyenv) elvis@perlmutter> despacktivate
(.spack-pyenv) elvis@perlmutter> spack env status
==> No active environment

(.spack-pyenv) elvis@perlmutter> spack spec zlib
Input spec
--------------------------------
zlib

Concretized
--------------------------------
zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=cray-sles15-zen3

We recommend you create a Spack environment in your user space to install your Spack stack. This can be done using spack env create command using the -d option to specify a directory. This will create a directory with an empty spack.yaml. To activate the environment you can run spack env activate and name of directory.

(.spack-pyenv) elvis@perlmutter> spack env create -d $HOME/e4s-22.05-demo
==> Created environment in /global/homes/s/elvis/e4s-22.05-demo
==> You can activate this environment with:
==>   spack env activate /global/homes/s/elvis/e4s-22.05-demo

(.spack-pyenv) elvis@perlmutter> spack env activate $HOME/e4s-22.05-demo

(.spack-pyenv) elvis@perlmutter> ls -l $HOME/e4s-22.05-demo
total 1
-rw-rw---- 1 elvis elvis 199 Jul 19 12:10 spack.yaml

You can retrieve the site scope via spack config --scope site get packages.

This Spack instance is configured to install packages in your HOME directory, the Spack site configuration can be found in this directory $SPACK_ROOT/etc/spack/ which applies configuration to all users but this can be overridden in your spack.yaml. Take note of the following settings below for build stage, and installation tree. Note that HOME directory is shared between Cori and Perlmutter, therefore we organize user installation by system. We recommend you adopt a similar naming scheme if you plan on building Spack packages on both systems.

(.spack-pyenv) elvis@perlmutter> cat $SPACK_ROOT/etc/spack/config.yaml
config:
  build_stage: $HOME/spack-workspace/perlmutter/build_stage
  misc_cache: $HOME/spack-workspace/perlmutter/misc_cache
  install_tree:
    root: $HOME/spack-workspace/perlmutter/software
  source_cache: $HOME/.spack/cache

Shown below is the site configuration for modules.yaml, we have configured the module root to be $HOME/spack-workspace/perlmutter/modules where TCL modules will be generated. Furthermore we have customized the TCL module setting when generating modules. You are welcome to override these configuration in your spack.yaml.

(.spack-pyenv) elvis@perlmutter> cat $SPACK_ROOT/etc/spack/modules.yaml
modules:
  default:
    roots:
      tcl: $HOME/spack-workspace/perlmutter/modules
    tcl:
      naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
      all:
        conflict:
        - '{name}'
        environment:
          set:
            '{name}_ROOT': '{prefix}'

Now let's add the following package in your spack.yaml and try building it via Spack. We will build papi. Shown below is the content of spack.yaml, please copy this content to your spack.yaml.

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs:
  - papi
  view: true

Note we don't provide a buildcache so papi will be built from source. Now let's install all packages in your environment by running spack install.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack install
==> Starting concretization
==> Environment concretized in 13.19 seconds.
==> Concretized papi
 -   s2y4nrv  papi@6.0.0.1%gcc@11.2.0~cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools arch=cray-sles15-zen3

==> Installing environment /global/homes/s/elvis/e4s-22.05-demo
==> Installing papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> No binary for papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af found: installing from source
==> Using cached archive: /global/homes/s/elvis/.spack/cache/_source-cache/archive/3c/3cd7ed50c65b0d21d66e46d0ba34cd171178af4bbf9d94e693915c1aca1e287f.tar.gz
==> No patches needed for papi
==> papi: Executing phase: 'autoreconf'
==> papi: Executing phase: 'configure'
==> papi: Executing phase: 'build'
==> papi: Executing phase: 'install'
==> papi: Successfully installed papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
  Fetch: 0.01s.  Build: 25.80s.  Total: 25.81s.
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af
==> Updating view at /global/homes/s/elvis/e4s-22.05-demo/.spack-env/view

We can find the root directory for papi by running the following. We see the package was installed in our home directory.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack location -i papi
/global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-s2y4nrvu6whr6hhgi63aa3nqwz2d35af

We can create the module tree and retrieve the location of our generated module as shown below.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack module tcl refresh --delete-tree -y
==> Regenerating tcl module files
(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> spack module tcl find --full-path papi
/global/homes/s/elvis/spack-workspace/perlmutter/modules/cray-sles15-zen3/papi/6.0.0.1-gcc-11.2.0-s2y4nrv

If you want to add this to MODULEPATH you can simply run the following:

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> module use $HOME/spack-workspace/perlmutter/modules/$(spack arch)

Now we see the modulefile papi/6.0.0.1-gcc-11.2.0-s2y4nrv is visible and can be loaded as any other software package.

(.spack-pyenv) elvis@perlmutter:~/e4s-22.05-demo> ml -t av papi
/global/homes/p/pratham/spack-workspace/perlmutter/modules/cray-sles15-zen3:
papi/6.0.0.1-gcc-11.2.0-s2y4nrv
/global/common/software/spackecp/perlmutter/e4s-22.05/modules/cray-sles15-zen3:
papi/6.0.0.1-gcc-11.2.0-cuda
papi/6.0.0.1-gcc-11.2.0
/opt/cray/pe/lmod/modulefiles/core:
papi/6.0.0.12
papi/6.0.0.14
papi/6.0.0.15

Binary Cache Mirror

We have setup a binary cache mirror named perlmutter-e4s-22.05 that can be added into your spack.yaml. This spack instance includes the binary cache mirror configuration.

You can check this by running spack mirror list to see the list of mirrors.

elvis@perlmutter>  spack mirror list
perlmutter-e4s-22.05    file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05
source_mirror           file:///global/cfs/cdirs/m3503/mirrors/source_mirror
spack-public            https://mirror.spack.io

Spack has support signing and verifying packages via GPG Key which is required when installing packages from the binary cache. To learn more about the binary cache, please see https://spack-tutorial.readthedocs.io/en/latest/tutorial_binary_cache.html#setting-up-a-binary-cache-mirror

First, you will need to install the GPG key locally, note this is only required once. Please run the following command.

elvis@perlmutter> spack buildcache keys -it
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/_pgp/B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F.pub
gpg: key 3BD6B0E9935AEB8F: public key "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: inserting ownertrust of 6

You can see list of gpg keys by running gpg -k make sure you see the following entry in the output.

pub   rsa2048 2022-02-23 [SC]
      B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
uid           [ultimate] GPG Key - e4s <shahzebsiddiqui@lbl.gov>
sub   rsa2048 2022-02-23 [E]

If you have successfully imported the gpg key you should be able to run spack buildcache list to see all the specs in the binary cache.

We have set the environment variable SPACK_GNUPGHOME which points to $HOME/.gnupg in case you run any commands like spack gpg which will typically fail since spack attempts to write GPG keys into the spack instance at $SPACK_ROOT/spack/opt/gpg which you won't have access to.

If you are still in your spack environment that you created in section Developer Workflow, let's install a package from the binary cache, you will notice that spack will install packages from the binary cache instead of building from source.

elvis@perlmutter> spack install raja@0.14.0~cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo
==> Installing pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0/cray-sles15-zen3-gcc-11.2.0-pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q.spack
==> Extracting pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q from binary cache
gpg: Signature made Fri 14 Oct 2022 08:12:17 AM PDT
gpg:                using RSA key B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
gpg: Good signature from "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" [ultimate]
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/pkgconf-1.8.0-nugfov2wsvpllczfy6ff7bytzt4oi34q
==> Installing berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3
==> Fetching file:///global/common/software/spackecp/mirrors/perlmutter-e4s-22.05/build_cache/cray-sles15-zen3/gcc-11.2.0/berkeley-db-18.1.40/cray-sles15-zen3-gcc-11.2.0-berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3.spack
==> Extracting berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3 from binary cache
gpg: Signature made Thu 13 Oct 2022 11:52:15 AM PDT
gpg:                using RSA key B5FDE18F615783AF078ED29C3BD6B0E9935AEB8F
gpg: Good signature from "GPG Key - e4s <shahzebsiddiqui@lbl.gov>" [ultimate]
[+] /global/homes/s/elvis/spack-workspace/perlmutter/software/cray-sles15-zen3/gcc-11.2.0/berkeley-db-18.1.40-hafeanvncfcwgstenlf5cc7ttyobmgt3

...