E4S 22.05 - Perlmutter¶
The e4s-22.05 stack is built using the Spack branch e4s-22.05 on Perlmutter using the gcc, nvhpc, and cce compilers. This stack can be loaded by running:
module load e4s/22.05
You can also load the same software stack via module load spack/e4s-22.05
Note
Going forward, we do not generate modules for Spack. In order to use E4S software you must use spack load to load a package in your user environment.
Here is a breakdown of the installed specs by spack environments:
| Spack Environments | Compiler | Root Specs | Implicit Specs | Total Specs |
|---|---|---|---|---|
gcc | gcc@11.2.0, gcc@10.3.0 | 55 | 262 | 317 |
cuda | gcc@11.2.0 | 17 | 97 | 114 |
nvhpc | nvhpc@22.5 | 6 | 19 | 25 |
cce | cce@15.0.0 | 8 | 64 | 72 |
Spack Configuration¶
When you load the e4s/22.05 module you will see several environments which will enable access to one of the sub-stacks. To see a list of environments you can run the following:
elvis@perlmutter> spack env list
==> 4 environments
cce cuda gcc nvhpc
You can run spack env status to retrieve your active environment. The message below indicates you are not in any active environment therefore you can't access the software stack.
elvis@perlmutter> spack env status
==> No active environment
For reference, please see Environments Tutorial for further assistance.
For instance, if you want to access the nvhpc environment you can do the following.
elvis@perlmutter> spack env activate -V nvhpc
elvis@perlmutter> spack env status
==> In environment nvhpc
The Spack configuration is accessible on the filesystem by changing directories into the root of the environment. For example, if you want to find the spack configuration for the nvhpc environment, you can run the following:
elvis@perlmutter> spack cd -e nvhpc
elvis@perlmutter> pwd && ls -l
/global/common/software/spackecp/perlmutter/e4s-22.05/78535/spack/var/spack/environments/nvhpc
total 56
-rw-rw-r-- 1 e4s spackecp 48395 Jan 11 13:51 spack.lock
-rw-rw-r-- 1 e4s spackecp 8172 Jan 11 13:50 spack.yaml
The file spack.yaml contains the configuration used to build the environments. You can find all of our spack configurations in the Spack Infrastructure project.
Overview¶
You can see a list of all available packages via spack find. Shown below is breakdown of packages by environments
Preview of the gcc spack environment
elvis@perlmutter> spack -e gcc find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%gcc@11.2.0 +blosc+bzip2~cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release /wlbzh44
amrex@22.05%gcc@11.2.0 ~amrdata~cuda~eb~fortran~hdf5~hypre~ipo+linear_solvers+mpi~openmp~particles~petsc~pic~plotfile_tools~rocm~shared~sundials~tiny_profile build_type=RelWithDebInfo dimensions=3 precision=double /ofkinx3
butterflypack@2.1.1%gcc@11.2.0 ~ipo+shared build_type=RelWithDebInfo /4kznqwp
ccache@4.5.1%gcc@11.2.0 ~ipo build_type=RelWithDebInfo /ybl7xef
cdo@2.0.5%gcc@11.2.0 ~curl~external-grib1+fftw3+hdf5+libxml2~magics+netcdf+openmp+proj+szip+udunits2 grib2=eccodes /mifq7z4
conduit@0.8.3%gcc@11.2.0 ~adios+blt_find_mpi~doc~doxygen+fortran+hdf5+hdf5_compat~ipo+mpi+parmetis~python+shared~silo+test~zfp build_type=RelWithDebInfo /wazineh
dyninst@12.1.0%gcc@11.2.0 ~ipo+openmp~stat_dysect~static build_type=RelWithDebInfo /imsyy3c
fortrilinos@2.0.0%gcc@11.2.0 +hl~ipo+shared build_type=RelWithDebInfo /gr4hjrx
gasnet@2022.3.0%gcc@11.2.0 ~cuda~debug~rocm conduits=smp /bwmz3br
gawk@5.1.1%gcc@11.2.0 ~nls /44kt3vx
git@2.35.3%gcc@11.2.0 +man+nls+perl+subtree~svn~tcltk /ubv32cm
gmake@4.3%gcc@11.2.0 ~guile+nls /btwzv5s
gnuplot@5.4.3%gcc@11.2.0 +X+cairo+gd+libcerf~pbm~qt~wx patches=ad89f23 /bsmjqym
grads@2.2.1%gcc@11.2.0 +geotiff+shapefile /lbkrgcs
gromacs@2021.5%gcc@11.2.0 ~blas~cuda~cycle_subcounters~double+hwloc~ipo~lapack~mdrun_only+mpi~nosuffix~opencl+openmp~plumed~relaxed_double_precision+shared~sycl build_type=RelWithDebInfo /oi4vmzo
gsl@2.7%gcc@11.2.0 ~external-cblas /fhx3zdz
hdf5@1.10.7%gcc@11.2.0 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /6opvtfc
heffte@2.2.0%gcc@11.2.0 ~cuda+fftw~fortran~ipo~magma~mkl~python~rocm+shared build_type=RelWithDebInfo /atk3gky
hpctoolkit@2022.04.15%gcc@11.2.0 ~all-static~cray~cuda~debug~level_zero~mpi+papi~rocm+viewer /aijvids
hpx@1.7.1%gcc@11.2.0 ~async_cuda~async_mpi~cuda~examples~generic_coroutines~ipo~rocm~tools build_type=RelWithDebInfo cxxstd=17 instrumentation=none malloc=tcmalloc max_cpu_count=64 networking=mpi /yqlat2t
hypre@2.24.0%gcc@11.2.0 ~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory /szss2c5
kokkos@3.6.00%gcc@11.2.0 ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper build_type=RelWithDebInfo std=14 /3b4inm2
kokkos-kernels@3.6.00%gcc@11.2.0 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /ss2tvzi
lammps@20220107%gcc@11.2.0 ~asphere~body~class2~colloid~compress~coreshell~cuda~cuda_mps~dipole~exceptions+ffmpeg~granular~ipo+jpeg+kim~kokkos~kspace~latte+lib~manybody~mc~meam~misc~mliap~molecule+mpi~mpiio~opencl+openmp~opt~peri+png~poems~python~qeq~replica~rigid~shock~snap~spin~srd~user-adios~user-atc~user-awpmd~user-bocs~user-cgsdk~user-colvars~user-diffraction~user-dpd~user-drude~user-eff~user-fep~user-h5md~user-lb~user-manifold~user-meamc~user-mesodpd~user-mesont~user-mgpt~user-misc~user-mofff~user-netcdf~user-omp~user-phonon~user-plumed~user-ptm~user-qtb~user-reaction~user-reaxc~user-sdpd~user-smd~user-smtbq~user-sph~user-tally~user-uef~user-yaff~voronoi build_type=RelWithDebInfo /h34ece6
libquo@1.3.1%gcc@11.2.0 /jdmo2xh
metis@5.1.0%nvhpc@22.5 ~gdb~int64~real64+shared build_type=Release patches=4991da9 /qxtdo74
nano@4.9%gcc@11.2.0 /3ajsqyz
nccmp@1.9.0.1%gcc@11.2.0 ~ipo build_type=RelWithDebInfo /svlmona
ncl@6.6.2%gcc@11.2.0 ~gdal~hdf4+openmp+triangle+udunits2 patches=64f3502,a2f7ac8,a612d41 /zmspv5c
nco@5.0.1%gcc@11.2.0 ~doc /t4avmjn
ncview@2.1.8%gcc@11.2.0 /5ujvhbd
nwchem@7.0.2%gcc@11.2.0 ~mpipr~openmp /hhxzzid
openblas@0.3.20%gcc@11.2.0 ~bignuma~consistent_fpcsr~ilp64+locking+pic+shared symbol_suffix=none threads=openmp /lrsjsel
openpmd-api@0.14.4%gcc@11.2.0 ~adios1+adios2+hdf5~ipo+mpi~python+shared build_type=RelWithDebInfo /q4ob2r7
parallel@20210922%gcc@11.2.0 /arb2fpm
parmetis@4.0.3%gcc@11.2.0 ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f89253,50ed208,704b84f /jdqvpwn
pdt@3.25.1%gcc@11.2.0 ~pic /4wzbhsa
petsc@3.17.1%gcc@11.2.0 ~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C /z7wbk6n
plasma@21.8.29%gcc@11.2.0 ~ipo~lua+shared build_type=RelWithDebInfo /sht5ri2
py-libensemble@0.9.1%gcc@10.3.0 ~deap~mpi~mpmath~nlopt~petsc4py~pyyaml~scipy~tasmanian /xy4w2mx
py-warpx@22.05%gcc@11.2.0 +mpi /nteuogz
py-warpx@22.05%gcc@11.2.0 +mpi /rmebedc
py-warpx@22.05%gcc@11.2.0 +mpi /atxm4s3
qthreads@1.16%gcc@11.2.0 +hwloc~spawn_cache+static scheduler=distrib stack_size=4096 /xsyqv4g
quantum-espresso@7.0%gcc@11.2.0 +cmake~elpa~environ+epw~gipaw~ipo~libxc+mpi~openmp+patch~qmcpack+scalapack build_type=RelWithDebInfo hdf5=none /olvwavb
raja@0.14.0%gcc@11.2.0 ~cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo /idylhfo
slate@2021.05.02%gcc@11.2.0 ~cuda~ipo+mpi+openmp~rocm+shared build_type=RelWithDebInfo /eh6rahv
slepc@3.17.1%gcc@11.2.0 +arpack~blopex~cuda~rocm /uetioli
subversion@1.14.1%gcc@11.2.0 +nls~perl~serf /fg7keqv
sundials@6.2.0%gcc@11.2.0 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /kfepsit
superlu@5.3.0%gcc@11.2.0 ~ipo+pic build_type=RelWithDebInfo /ziz42oa
tasmanian@7.7%gcc@11.2.0 ~blas~cuda~fortran~ipo~magma~mpi+openmp~python~rocm~xsdkflags build_type=Release /4yefyvt
upcxx@2022.3.0%gcc@11.2.0 ~cuda~gasnet~mpi~rocm cross=none /yjru4k5
vtk-m@1.7.1%gcc@11.2.0 ~64bitids+ascent_types~cuda+doubleprecision~ipo~kokkos~logging~mpi+openmp+rendering~rocm~shared~tbb~testlib~virtuals build_type=Release /zvu6y2s
xterm@353%gcc@11.2.0 /s6hwjyj
Preview of the nvhpc environment
elvis@perlmutter> spack -e nvhpc find --format "{name}@{version}%{compiler} {variants} /{hash:7}"
hdf5@1.10.7%nvhpc@22.5 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /q7pqnux
kokkos@3.6.00%nvhpc@22.5 ~aggressive_vectorization~compiler_warnings~cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl+openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning~wrapper build_type=RelWithDebInfo std=14 /lbfztq6
kokkos-kernels@3.6.00%nvhpc@22.5 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /qspw732
sundials@6.2.0%nvhpc@22.5 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /4bb7vwm
superlu@5.3.0%nvhpc@22.5 ~ipo+pic build_type=RelWithDebInfo /gs4k3my
zfp@0.5.5%nvhpc@22.5 ~aligned~c~cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo /a3kwh2u
Preview of the cce environment
elvis@perlmutter> spack -e cce find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%cce@15.0.0 +blosc+bzip2~cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release /75i5mf3
hdf5@1.10.7%cce@15.0.0 ~cxx+fortran+hl~ipo~java+mpi+shared~szip~threadsafe+tools api=v18 build_type=RelWithDebInfo patches=2a1e311 /2hv5uoq
hypre@2.24.0%cce@15.0.0 ~complex~cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory /ngi2ghh
kokkos-kernels@3.6.00%cce@15.0.0 ~blas~cblas~cublas~cuda~cusparse~ipo~lapack~lapacke~mkl+openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /aclkqoy
petsc@3.17.1%cce@15.0.0 ~X~batch~cgns~complex~cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared~strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C /f5232qk
sundials@6.2.0%cce@15.0.0 +ARKODE+CVODE+CVODES+IDA+IDAS+KINSOL~cuda+examples+examples-install~f2003~fcmix+generic-math~hypre~int64~ipo~klu~lapack~magma~monitoring+mpi~openmp~petsc~profiling~pthread~raja~rocm+shared+static~superlu-dist~superlu-mt~sycl~trilinos build_type=RelWithDebInfo cstd=99 cxxstd=14 logging-level=0 precision=double /vigxlll
superlu@5.3.0%cce@15.0.0 ~ipo+pic build_type=RelWithDebInfo /x7icfwr
superlu-dist@7.2.0%cce@15.0.0 ~cuda~int64~ipo+openmp~rocm+shared build_type=RelWithDebInfo patches=8da9e21 /5kwghlm
Preview of the cuda environment
elvis@perlmutter> spack -e cuda find -x --format "{name}@{version}%{compiler} {variants} /{hash:7}"
adios2@2.8.0%gcc@11.2.0 +blosc+bzip2+cuda~dataman~dataspaces+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release cuda_arch=80 /df2vmcb
hpctoolkit@2022.04.15%gcc@11.2.0 ~all-static~cray+cuda~debug~level_zero+mpi~papi~rocm+viewer /ntu5gi2
hypre@2.24.0%gcc@11.2.0 ~complex+cuda~debug+fortran~gptune~int64~internal-superlu~mixedint+mpi+openmp~rocm+shared+superlu-dist~unified-memory cuda_arch=80 /bonsnuz
kokkos@3.6.00%gcc@11.2.0 ~aggressive_vectorization~compiler_warnings+cuda~cuda_constexpr~cuda_lambda~cuda_ldg_intrinsic~cuda_relocatable_device_code~cuda_uvm~debug~debug_bounds_check~debug_dualview_modify_check~deprecated_code~examples~explicit_instantiation~hpx~hpx_async_dispatch~hwloc~ipo~memkind~numactl~openmp~pic+profiling~profiling_load_print~pthread~qthread~rocm+serial+shared~sycl~tests~tuning+wrapper build_type=RelWithDebInfo cuda_arch=80 std=14 /lonrnyk
kokkos-kernels@3.6.00%gcc@11.2.0 ~blas~cblas~cublas+cuda~cusparse~ipo~lapack~lapacke~mkl~openmp~pthread~serial+shared~superlu build_type=RelWithDebInfo cuda_arch=80 execspace_cuda=auto execspace_openmp=auto execspace_serial=auto execspace_threads=auto layouts=left memspace_cudaspace=auto memspace_cudauvmspace=auto offsets=int,size_t ordinals=int scalars=double /swui7am
lammps@20220107%gcc@11.2.0 ~asphere~body~class2~colloid~compress~coreshell+cuda+cuda_mps~dipole~exceptions+ffmpeg~granular~ipo+jpeg+kim~kokkos~kspace~latte+lib~manybody~mc~meam~misc~mliap~molecule+mpi~mpiio~opencl+openmp~opt~peri+png~poems~python~qeq~replica~rigid~shock~snap~spin~srd~user-adios~user-atc~user-awpmd~user-bocs~user-cgsdk~user-colvars~user-diffraction~user-dpd~user-drude~user-eff~user-fep~user-h5md~user-lb~user-manifold~user-meamc~user-mesodpd~user-mesont~user-mgpt~user-misc~user-mofff~user-netcdf~user-omp~user-phonon~user-plumed~user-ptm~user-qtb~user-reaction~user-reaxc~user-sdpd~user-smd~user-smtbq~user-sph~user-tally~user-uef~user-yaff~voronoi build_type=RelWithDebInfo cuda_arch=80 /uqdra2u
likwid@5.2.1%gcc@11.2.0 +cuda+fortran /wst4i5t
papi@6.0.0.1%gcc@11.2.0 +cuda+example~infiniband~lmsensors~nvml~powercap~rapl~rocm~rocm_smi~sde+shared~static_tools /zy7rjq4
parmetis@4.0.3%gcc@11.2.0 ~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f89253,50ed208,704b84f /jdqvpwn
pdt@3.25.1%gcc@11.2.0 ~pic /4wzbhsa
petsc@3.17.1%gcc@11.2.0 ~X~batch~cgns~complex+cuda~debug+double~exodusii~fftw+fortran~giflib+hdf5~hpddm~hwloc+hypre~int64~jpeg~knl~kokkos~libpng~libyaml~memkind+metis~mkl-pardiso~mmg~moab~mpfr+mpi~mumps+openmp~p4est~parmmg~ptscotch~random123~rocm~saws~scalapack+shared+strumpack~suite-sparse+superlu-dist~tetgen~trilinos~valgrind clanguage=C cuda_arch=80 /nb2ei2k
raja@0.14.0%gcc@11.2.0 +cuda+examples+exercises~ipo+openmp~rocm+shared~tests build_type=RelWithDebInfo cuda_arch=80 /gueomxk
slate@2021.05.02%gcc@11.2.0 +cuda~ipo+mpi+openmp~rocm+shared build_type=RelWithDebInfo cuda_arch=80 /bdcxawb
slepc@3.17.1%gcc@11.2.0 +arpack~blopex+cuda~rocm cuda_arch=80 /lmzfg45
strumpack@6.3.1%gcc@11.2.0 ~butterflypack+c_interface~count_flops+cuda~ipo+mpi+openmp+parmetis~rocm~scotch+shared~slate~task_timers+zfp build_type=RelWithDebInfo cuda_arch=80 /pocpro4
tau@2.31.1%gcc@11.2.0 ~adios2+binutils~comm~craycnl+cuda+elf+fortran~gasnet+io~level_zero+libdwarf+libunwind~likwid+mpi~ompt~opari~opencl~openmp+otf2+papi+pdt~phase~ppc64le~profileparam+pthreads~python~rocm~rocprofiler~roctracer~scorep~shmem~sqlite~x86_64 /5uqetmp
zfp@0.5.5%gcc@11.2.0 ~aligned~c+cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=80 /k4cqxwj
Shown below are a few useful spack find commands you can use to filter output.
All root specs built with the NVHPC compiler
elvis@perlmutter> spack find -x --format "{name}" %nvhpc
hdf5 kokkos kokkos-kernels sundials superlu zfp