Python is an interpreted general-purpose high-level programming language. You can use Anaconda Python on Cori through software environment modules. Do not use the system-provided Python
Anaconda Python is a platform for large-scale data processing, predictive analytics, and scientific computing. It includes hundreds of open source packages and Intel MKL optimizations throughout the scientific Python stack. Anaconda provides the
conda command-line tool for managing packages, but also works well with
pip. The Anaconda distribution also exposes access to the Intel Distribution for Python.
Python 3 is the default Python module. To load it, type:
module load python
The default is
python/3.7-anaconda-2019.10 so only
module load python is necessary to use it.
When you load a Python module you are placed into its default Anaconda "root" environment. This may be sufficient for your needs. NERSC can install Anaconda packages into the root environment upon request subject to review of broad utility and maintainability. Contact us to find out about specific packages. In general we recommend users manage their own Python installations with "conda environments."
conda tool lets you build your own custom Python installation through "environments." Conda environments replace and surpass
virtualenv virtual environments in many ways. To create and start using a conda environment you can use
conda create. Specify a name for the environment and at least one Python package to install. In particular you should specify which Python interpreter you want installed. Otherwise
conda may make a decision that surprises you.
module load python conda create -n myenv python=3 numpy [installation outputs] source activate myenv
Should I use
source activate\deactivate or
You may have noticed messages like:
To activate this environment, use $ conda activate <environment> To deactivate an active environment, use $ conda deactivate
source activate <environment>or
source activate && conda activate <environment>
conda deactivateinstead of
Long answer: Using
conda activate <environment> will fail with a message to run conda init
conda init will try to modify the shell initialization script (.bashrc for bash), which is owned by a restricted user, not by you. Running
source activate (no environment specified) will effectively emulate the functionality of
conda init <shell>. Once, conda is activated, either by
source activate && conda activate <environment> or
source activate <environment>, it is then possible (and recommended) to use the modern
conda deactivate to deactivate the conda environment.
Part of the issue is that conda does not provide a way for us to edit this configuration. The other part is how we have configured our own system. We are hoping that one or both of these issues can be resolved soon so we can switch to the more modern
Before it installs anything
conda create will show you what package management steps it will take and where the installation will go. You will be asked for confirmation before installation proceeds.
The Life You Save May Be Your Own
Make it a habit to actually review
conda tool reports and not just blithely punch the "y" key to approve create/install/update actions. Verify the installation is going where you think it should. Make sure any proposed package downgrades are acceptable.
Once the environment is created you need to switch to it using
source activate. After activation the name of the environment will appear in your prompt.
To leave an environment behind use:
Bad News for csh/tcsh Users
If you use csh or tcsh you will not be able to use the
source activate syntax. For csh users this is a shortcoming of the conda tool. There are workarounds available on the web that work to varying degrees. (We often find users are able to switch to /bin/bash without much difficulty, that is one solution.)
If you are a csh user and you do not need to install or manage packages once a conda environment has been provisioned, you can simply set
PATH to point to the path of the Python interpreter in the environment.
You can find packages and install them into your own environments easily.
conda search scipy [list of available versions of scipy] conda install scipy
conda search fails to identify your desired package it may still be installed via
pip can be used in conda environments.
Use conda to install pip into your environment
pip in your own environment you may need to
conda install pip. Verify whether you need to by typing "
which pip" at the command line. If the path returned looks like
/usr/common/software/python/.../bin/pip then do
conda install pip.
If you consider
pip a last resort you may want to search non-default channels for builds of the package you want. The syntax for that is a little different:
anaconda search -t conda <package-name> [list of channels providing the package] conda install -c <channel-name> <package-name>
Finally you can install packages "from source" and in some cases this is recommended. In particular any package that depends on the Cray programming environment should be installed this way. For Python this usually boils down to
h5py with MPI-IO support.
Run serial Python scripts on a login node, or on a compute node in an interactive session (started via salloc) or batch job (submitted via sbatch) as you normally would in any Unix-like environment. On login nodes, please be mindful of resource consumption since those nodes are shared by many users at the same time.
Parallel Python scripts launched in an interactive (salloc) session or batch job (sbatch), such as those using MPI via the mpi4py module, must use srun to launch:
srun -n 64 python ./hello-world.py
Please see this page for more information about using mpi4py.
Please see this page for more information about using h5py MPI-IO.
Collaborations, projects, or experiments may wish to install a shareable, managed Python stack to
/global/common/software independent of the NERSC modules. You are welcome to use the Anaconda installer script for this purpose. In fact you may want to consider the more "stripped-down" Miniconda installer as a starting point. That option allows you to start with only the bare essentials and build up. Be sure to select Linux version in either case! For instance:
https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh -b \ -p /global/common/software/myproject/env [installation messages] source /global/common/software/myproject/env/bin/activate conda install <only-what-my-project-needs>
You can customize the path with the
-p argument. Ihe installation above would go to
$HOME/miniconda3 without it.
When using your own Anaconda/Miniconda installation be sure not to load any NERSC-provided Python modules. Also take care to consider the
PYTHONSTARTUP environment variable which you may wish to unset altogether. It is mainly relevant to the system Python we advise against using.
Note that to activate the root environment, technically you should use the
source shell command. Setting
PATH to the root environment
bin directory works but the source/conda tool does more than that.
End-of-Life for Python 2¶
Python 2 reached end-of-life on January 1, 2020. After its final release, Python 2 ceases to exist as an active project: No development, no bug fixes, no patches, etc. This is important because users must actively transition to Python 3, which is not backward-compatible with Python 2.
Developers of many packages including NumPy, SciPy, Matplotlib, Pandas, and Scikit-Learn pledged to drop support for Python 2 "no later than 2020." You can expect support for all Python 2 libraries to continue to wither away. Using Python 2 past end of life is a risk as new issues will likely go unaddressed by developers. You may already have noticed deprecation warnings from your Python applications' outputs; do not ignore these warnings! You should develop a migration plan soon if you don't have one already.
At the Python 3 Statement website, there are a few links under the “Why?” section that may be helpful to you in preparing your migration plan and easing the transition from Python 2 to Python 3. These seem especially helpful:
If you have any questions, please let us know via a ticket at help.nersc.gov.