The Virtual Environment (virtualenv) framework allows for creation of isolated Python installations. It is the traditional tool developers use to quickly configure custom python installations and switch between functioning environments for distinct projects. When used with pip, it is very easy to build environments and distribute “requirements” files for rebuilding these environments on different systems. The additional library virtualenvwrapper facilitates installation management and switching between environments.

Environment Creation and Activation

# Load the python module for the version you wish to use.
scc1$ module load python3/3.7.7

# Create virtual environment.
scc1$ virtualenv myenv
New python executable in ~/myenv/myenv/bin/python
Installing setuptools, pip, wheel...done.

# Activate it
scc1$ source myenv/bin/activate
(mynewenv) scc1$  

# Install packages into this virtualenv (e.g. "htseq")
(mynewenv) scc1$ pip install htseq
Collecting htseq
Collecting numpy
Collecting pysam
Installing collected packages: numpy, pysam, htseq
Successfully installed htseq-0.12.4 numpy-1.19.1 pysam-0.16.0.1

# Make sure it works
(mynewenv) scc1$ which python
~/mynewenv/bin/python
(mynewenv) scc1$ python
Python 3.7.7 (default, May 21 2020, 14:57:43) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>> import HTSeq
>>  
>> exit()
(mynewenv) scc1$ which htseq-count 
~/mynewenv/bin/htseq-count

# Exit/Deactivate the virtual environment
(mynewenv) scc1$ deactivate
scc1$

Exporting and Distributing Environments

# Load the python module for the version you wish to use.
scc1$ module load python3/3.7.7
scc1$ source mynewenv/bin/activate
(mynewenv) scc1$ 

# See what is installed in this virtualenv.
(mynewenv) scc1$ pip list --local
Package    Version
---------- --------
HTSeq      0.12.4
numpy      1.19.1
pip        20.1.1
pysam      0.16.0.1
setuptools 46.4.0
wheel      0.34.2
# Export list of packages in pip “requirements” format (you can give this to someone).
(mynewenv) scc1$ pip freeze --local > requirements.txt
(mynewenv) scc1$ cat requirements.txt
HTSeq==0.12.4
numpy==1.19.1
pysam==0.16.0.1
...

# Exit/Deactivate current virtual environment.
(mynewenv) scc1$ deactivate
scc1$ 

# Build new virtualenv.
scc1$ virtualenv mysecondenv
New python executable in /usr3/bustaff/cjahnke/mysecondenv/bin/python
Installing setuptools, pip, wheel...done.
scc1$ source mysecondenv/bin/activate
(mysecondenv) scc1$ 

# Install everything in the requirements file.
(mysecondenv) scc1$ pip install -r requirements.txt
Collecting htseq
Collecting numpy
Collecting pysam
Installing collected packages: numpy, pysam, htseq
Successfully installed htseq-0.12.4 numpy-1.19.1 pysam-0.16.0.1

# Test.
(mysecondenv) scc1$ which htseq-count
~/mysecondenv/bin/htseq-count
(mysecondenv) scc1$ python
Python 3.7.7 (default, May 21 2020, 14:57:43) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>> import HTSeq
>>

Virtual Environment Wrapper (virtualenvwrapper)

The virtualenvwrapper package is a set of extensions to the virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies.

  • Organizes all of your virtual environments in one place.
  • Wrappers for managing your virtual environments (create, delete, copy).
  • Use a single command to switch between environments.
  • Tab completion for commands that take a virtual environment as argument.
  • User-configurable hooks for all operations.
  • Plugin system for more creating shareable extensions.

Official Documentation: https://virtualenvwrapper.readthedocs.io/en/latest/

 

Using virtualenvwrapper

Decide on a version of python and the location for your virtual environments. Load the python module and set the “$WORKON_HOME” environment variable to reflect your personal location for virtual environments. The virtualenvwrapper scripts use the $WORKON_HOME variable frequently and it will need to be set anytime you want to use the virtualenvwrapper scripts. In this example, I will use python3/3.7.7 and place my virtualenvs in /projectnb/scv/cjahnke/virtenvs.

Note: The recommended location for virtualenvs is in the /projectnb space of any project. The home directory is a not recommended due to strict space and file number quotas.
[cjahnke@scc4 virtenvs]$ pwd
/projectnb/scv/cjahnke/virtenvs
[cjahnke@scc4 virtenvs]$ module load python3/3.7.7
[cjahnke@scc4 virtenvs]$ export WORKON_HOME=/projectnb/scv/cjahnke/virtenvs

 

Source the virtualenvwrapper.sh script to setup the virtualenvwrapper scripts and enable the new features. The virtualenvwrapper.sh script is added to your $PATH when the python/2.7.13 module is loaded.

[cjahnke@scc4 virtenvs]$ source virtualenvwrapper.sh
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/premkproject
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/postmkproject
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/initialize
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/predeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/postdeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/preactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/postactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/get_env_details

 

Make a new environment using the mkvirtualenv command. This will create the space and automatically enable it. When an environment is enabled, you will see the pre-prompt with the virtual environment name in parentheses.

[cjahnke@scc4 virtenvs]$ mkvirtualenv env1
New python executable in /projectnb/scv/cjahnke/virtenvs/env1/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env1/bin/predeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env1/bin/postdeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env1/bin/preactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env1/bin/postactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env1/bin/get_env_details
(env1) [cjahnke@scc4 virtenvs]$

 

Install desired packages into this environment. As an example, we will use pip to install ‘htseq’, a bioinformatics package that has some dependencies.

(env1) [cjahnke@scc4 virtenvs]$ pip install htseq
Collecting htseq
Collecting numpy (from htseq)
Using cached numpy-1.19.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting pysam>=0.9.0 (from htseq)
Using cached pysam-0.16.0.1-cp37-cp37m-manylinux1_x86_64.whl
Installing collected packages: numpy, pysam, htseq
Successfully installed htseq-0.12.4 numpy-1.19.1 pysam-0.16.0.1
(env1) [cjahnke@scc4 virtenvs]$

Use the mkvirtualenv command to create another environment. This environment will have distinct package set from the first. Again, we see the new environment enabled automatically.

(env1) [cjahnke@scc4 virtenvs]$ mkvirtualenv env2
New python executable in /projectnb/scv/cjahnke/virtenvs/env2/bin/python
Installing setuptools, pip, wheel...done.
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env2/bin/predeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env2/bin/postdeactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env2/bin/preactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env2/bin/postactivate
virtualenvwrapper.user_scripts creating /projectnb/scv/cjahnke/virtenvs/env2/bin/get_env_details
(env2) [cjahnke@scc4 virtenvs]$

 

Use the workon command to easily switch between environments. Notice the pre-prompt changes to indicate the current virtualenv.

(env2) [cjahnke@scc4 virtenvs]$ workon env1
(env1) [cjahnke@scc4 virtenvs]$

 

Use lsvirtualenv to list available environments.

(env1) [cjahnke@scc4 virtenvs]$ lsvirtualenv -b
mynewenv1
mynewenv2

 

There are many additional conveniences built into the virtualenvwrapper, please review the official documentation.