Virtual environments in Python
The venv module in Python creates a light virtual environment that allows users to isolate specific versions of the Python interpreter and software libraries for their projects. The venv module comes with the standard library of Python and does not require a separate installation. Venv is ideal for project-based workflows.
There are a few advantages to working with virtual environments:
- They encapsulate project specific dependencies to prevent any updates to packages to break codes for different projects
- They prevent dependency conflicts
- They allow for reproducibility
module load python/version
and then create your virtual environment with venv.Create a virtual environment with venv
To create a virtual environment with venv
, enter the following command in your command line:
python -m venv /PATH/TO/NEW/ENVIRONMENT
The virtual environment is created in the /PATH/TO/MY/NEW/ENVIRONMENT
directory, so be sure to replace /PATH/TO/MY/NEW/ENVIRONMENT
with directory path that you want your environment to be created in. This command also creates any intermediary directories if they do not exist already.
python -m venv -h
.Activate the environment
To activate the new environment, use the source
command:
source /PATH/TO/MY/ENVIRONMENT/bin/activate
Be sure to replace PATH/TO/MY/ENVIRONMENT
with the actual path to the directory of your virtual environment. The terminal displays the name of your environment in parenthesis, which means your environment is safely activated.
Install packages
Within your virtual environment, you can use common installation tools such as pip to install packages into your virtual environment.
To download any packages that you need for your environment, use the pip
command followed by the package(s) you want to install. The following example installs the NumPy and Pandas packages:
pip install numpy pandas
These packages are installed in the venv environment
--without-pip
option.pip freeze > requirements.txt
.Deactivate the virtual environment
To deactivate the virtual environment, use the deactivate
command:
deactivate
Move your venv virtual environment
The venv environment itself is not portable, but it can be easily recreated in another location from the requirements.txt file. The following steps will show you how to create a new environment called MY_NEW_ENV and download the packages from the requirements.txt file.
If you haven’t already, create a requirements.txt file using the following command:
pip freeze > requirements.txt
To load the correct python version, use the following command:
module load python/3.11.0
To create the venv environment, use the following command:
python -m venv MY_NEW_ENV
To install packages from the requirements.txt file, use the following command:
pip install -r requirements.txt
To activate the environment, use the following command:
source MY_NEW_ENV/bin/activate
Submit a slurm job with venv
To submit a sbatch
job using a venv environment, you can source
the environment at the top of the sbatch script.
The following example assumes there is a script called testscript.py that needs to be submitted. This example script prints the versions of the packages we have downloaded in the Move your venv virtual environment example environment called MY_NEW_ENV.
testscript.py
import numpy as np
import pandas as pd
import sys
print(f"Python version = {sys.version}")
print(f"Numpy version = {np.version.version}")
print(f"Pandas version = {pd.__version__}")
To submit this script using a venv environment, create the following script and save it as main.sh.
main.sh
#!/usr/bin/bash
#SBATCH --partition=cpu-preempt # Partition (queue) name
#SBATCH --ntasks=1 # Number of CPU cores
#SBATCH --nodes=1 # Number of nodes
#SBATCH --mem=1gb # Job memory request
#SBATCH --time=00:01:00 # Time limit hrs:min:sec
#SBATCH --output=pyenv_test_%j.log # Standard output and error log
# Load venv env
source /PATH/TO/MY/ENVIRONMENT/bin/activate
# Run script
python testscript.py
Submit the main.sh script to slurm using the command sbatch main.sh
. The results are captured in the --output
file.
Other virtual environment software options
The following are some additional software options for creating and/or managing virtual environments.
- Virtualenv is the original and most popular tool to create virtual environments in Python. Venv started as a subset of virtualenv. The biggest advantage is that it works with old python versions.
- Virtualenvwrapper is a set of extension tools for managing multiple virtualenvs.
- Pipreqs is a helpful tool for creating requirements.txt files. While
pip freeze
only logs packages installed withpip
,pipreqs
will do a more comprehensive job in creating your requirements file and only includes packages that are actively used in your project.