Software Tools
Software Environments via Conda
Software Environments via Conda
HBSGrid administrators use conda to install R and Python environments that you can access via the software modules system. You can also create, use, and share your own conda environments, e.g., to facilitate reproduction of your analyses, to keep software requirements separate for different projects,or to install specialized software that is not already available on the HBSGrid.
Note: If working solo or needing only a few person modules/packages, we recommend instead using the standard Python / R installation methods, as outlined on our R and Python page. This page is intended for those working on multiple projects and would like to create separate environments with custom module installations, or for the reasons outlined previously.
Conda is one way regular users (i.e., those without administrator accounts) can install software on the HBSGrid. Other methods are documented on the installing software page.
The notes below give a quick introduction to using conda on the HBSGrid. For more details refer to the official conda documentation.
- Accessing Conda on the HBSGrid
-
You can load the conda package manager using the software modules system. Open a terminal on the HBSGrid and run
module load anaconda
You will need to do this after every login before working with conda environments.
Note: We strongly discourage you from including module loads and conda activate commands in your login scripts (
~/.bash_profile
and~/.bashrc
) as this could seriously interfere with future logins.After loading the anaconda module you can verify that it is available by running
conda --version
in the terminal. You can list the conda environments already installed with
conda env list
This command will list any you have created along with environments that are centrally installed on the HBSGrid cluster.
- Creating, Activating, and Modifying Conda Environments
-
Conda uses environments to keep different versions of programs separate. You can search for packages using
conda search
, e.g. to search for available python versions runconda search python
(If you want to search for packages with
python
anywhere in the name useconda search "*python*"
.) Note that while the examples here focus on python packages, conda is a general-purpose package manager that can install other kinds of software as well.You can create conda environments using the
conda create
command. For example, you can create a conda environment with python version 3.8 withconda create --name py38 python=3.8
In order to use this environment you must activate it:
conda activate py38
Once actived, you can run the software installed in your environment as usual, i.e. using
bsub
to start an interactive job. For example,bsub -Is python
will run on a compute node the python 3.8 version you installed a few steps prior as an interactive session with jobs defaults (if not specified, defaults are 1 core, 5 - 8 GB RAM, and the
long
orlong_int
queue).You can install additional packages into an existing environment using
conda install
. For example, to install the pandas python package into the py38 environment:conda install --name py38 pandas
- The Conda-Forge Repository Channel
-
The default conda repository channel contains thousands of packages, but you may find that it does not include a particular program you need for your research. In that case you can use the conda-forge repository, an even larger community-maintained collection of conda packages.
For example, if you need to extract data from tables in PDF files, you can create a conda environment and install the Camelot PDF table extraction tool:
conda create --name camelot --channel conda-forge camelot-py
Once created you can activate and run software in this environment using
conda activate
as before, i.e.,conda activate camelot
camelot --help - Sharing Conda Environments
-
If you are collaborating using shared project spaces on the HBSGrid you can create and activate conda environments using the
--prefix
option. For example, you can useconda create --prefix /export/projects/myproject/myenv
to create an environment in
/export/projects/myproject/
, andconda activate /export/projects/myproject/myenv
to activate it. Your collaborators with access to
/export/projects/myproject/
can also activate and use this environment withconda activate /export/projects/myproject/myenv
An alternative method of sharing conda environments that does not require shared directory access is described in the official conda documentation, where you can also find a wealth of other useful information about using conda.