R Python On The HBSGrid

Introduction

The HBSGrid uses a combination of Anaconda, miniConda (a slimmed down Anaconda), and software modules to provide feature-rich environments for Python and R. This page briefly discusses need-to-knows for each environment.

R

The HBSGrid offers multiple versions of R, including R 4.0.x (default) along with RStudio 1.3 (default). The default versions of R and RStudio are installed using the miniConda distribution and is available via wrapper scripts from the NoMachine GUI menus or terminal. These versions and other versions of R and RStudio can be accessed using the instructions on our software modules and accessing software via modules pages and used in combination with wrapper scripts in the terminal. All versions can be accessed via software modules and custom LSF submission scripts. Please note that R 3.5 cannot be used with RStudio 1.3; and likewise, R 3.6 and 4.0 cannot be used with RStudio 1.1. The software module system will prevent you from doing so.

Installing custom R packages

No special instructions are needed for this. Using the install.packages() command from within R or RStudio will download and install the specified packages in your home folder by default. If you are working on a team, you may consider using project-specific custom installations. For additional information, please contact RCS.

Python

We offer on the HBSGrid multiple flavors of Python: Python 3.8.x (default), Python 3.7.x, 3.6.x, and 2.7.x Python 3.8.x is installed via the Anaconda distribution and are available as defaults via wrapper scripts from the NoMachine GUI menus or terminal. Python can be accessed using the instructions on software modules and accessing software via modules pages and used in combination with wrapper scripts in the terminal. All versions can be accessed via software modules and custom LSF submission scripts. The AI/ML environment is Python 3.8 with Tensorflow-GPU, PyTorch, and Keras. Please note that each version of Python is incompatible with the other version and cannot be used in a given job script without loading a different software module. Note: If loading Python3 via software modules and one wishes to use the wrapper scripts, it important to continue to use python3-Xg or spyder3-Xg wrappers, as python-XG and spyder-XG still point to python2, the cluster wide default version of 'python'.

Installing custom Python modules

If you require a Python module that is not installed by default, you can install this yourself, and the module will be placed in a directory in your home folder. Due to the wrapper scripts that are installed on login nodes, this action must be performed via the back-end, compute nodes. 1. Set up an alias so it's easy to submit interactive jobs to back-end nodes: alias my_run="bsub -app python-5g -q short_int -Is" 2. Install your Python module by prefixing the install command with my_run: my_run python -m pip install --user SomeModule OR, If you are using Python3, use the python3 command instead: my_run python3 -m pip3 install --user SomeModule The modules will be placed in the directory $HOME/.local for use by your scripts and programs. If you are upgrading modules that are already installed centrally, insert --upgrade before the --user option.

Python and Conda Environments

If you are working on multiple projects and would like to create separate environments with custom module installations, you may consider using conda virtual environments. With a virtual environment you can update the packages for one project without disturbing the packages of your other projects. Our Conda Environmentsweb page has details on how to take advantage of this great toolset. In addition, it's important to note the following:

  • One no longer needs to install Anaconda/Miniconda in your home directory.

  • With

    software modules

    now on the HBSGrid,

    module load

    one of the

    conda

    , recent

    anaconda

    , or default

    anaconda

    module files to set up your terminal session for using

    conda (the package and environment manager)

    .

  • By default, new environments are placed in your home directory. Once you

    source activate

    or

    conda activate

    your environment, the execution

    PATH

    should point towards your local (home directory) install.

  • Environments now can be used with the

    wrapper scripts

    in NoMachine and from the command line, as the execution

    PATH

    points at and prioritizes either your activate environment or the central environment/installations (if activated via

    module load python/XXX)

    . See our

    Recent HBSGrid Improvements

    news posting.

Jupyter Notebooks

Please note that Jupyter Notebooks cannot be used on the HBSGrid due to security concerns. RCS is working with HBS IT and IT Security to investigate what options we have to provide this functionality. Although not ideal, the Python IDE Spyder, which is provided on the HBSGrid cluster, can be via "Run Selected Code" or "Run Current Line" menu items to emulate running a Jupyter Notebooks code cell. We'll update the community when we have more information.

Installed R Packages & Python Modules

Our base installations of R and Python have the packages and modules listed below as a part of the central installation. You can use the instructions above to install additional or updates ones in either your home folder (the default location) or in another location that you must manually specify.