Unity software overview

Ways to install packages

apt package manager

The Ubuntu system package manager apt downloads its packages pre-compiled from the Ubuntu repository. These are in standard locations like /usr/bin so that they are always found in your $PATH. This requires administrator access. To avoid conflicts, most software is only available in one version. As a rule, we only install basic support packages this way, not research software.

Environment modules

There are a wide variety of modules available with the module command. Most software requests can be found this way. The admins usually compile and install software in /modules/apps/ or in /modules/spack/packages/.

Relevant documentation:

• Introduction to environment modules
• How to use environment modules

Conda environments

The conda package manager allows users to compile software easily and without administrator privileges. Conda creates environments for a set of compatible software, and you can activate the environments as needed.

Relevant Documentation:

• Conda environments

Virtual environments in Python

The venv module in Python creates a light virtual environment that allows users to isolate specific versions of the Python interpreter and software libraries for their projects. It comes with the standard library of Python and does not require a separate installation. Venv is ideal for project-based workflows.

Relevant documentation:

• Virtual environments in python

R package management

Generally you can use R’s native install.packages() and it should work as intended. However, if you have external dependencies, using them with our standard modules r-rocker-ml-verse/*-apptainer may run into issues. You can build a container for it, using a recipe like this:

Bootstrap: docker
From: rocker/ml-verse:{{ R_VERSION }}

%arguments
    R_VERSION=4.4.0

%post
    apt-get -y update
    apt-get -y install jags

    # Install R packages
    Rscript -e 'install.packages("pak")'
    Rscript -e 'pak::pak(c("rjags", "jagsUI", "coda", "EnvStats", "sads"))'

And then build the container giving a destination image name (.sif) and the preceding definition file:

apptainer build --ignore-fakeroot-command my-r.sif my-r.def

Relevant Documentation:

• R

Containers: Docker, Singularity, and Apptainer

While Docker is not supported in an HPC environment, Apptainer provides nearly equivalent capability. It’s possible to pull an existing Docker image into a local Apptainer image file like this:

apptainer pull docker://ollama/ollama

If you need to install your own set of software, you can create and build your own container as show preceding in R Package Management

Building software from scratch

Building software from scratch is the most flexible method, but also the most involved. In some cases, it requires deep knowledge of the software in question, including its dependencies and other requirements.

While any well maintained software should provide some installation instructions, usually in a README or INSTALL file, there are many that assume the user has some knowledge of common build systems. This is particularly true for code that is mostly in C/C++ or Fortran.

The following content is a general guide. Please read it completely before starting, but be aware that a single guide can not capture all of the potential complexity.

Part 1: Obtaining the code

When fetching the code, we recommend using compressed release bundles instead of doing a git clone. Compressed release bundles are:

• Usually single files with an extension of .tar.gz, .tar.bz2, .tar.xz, or .zip.
• Often contain a “build ready” set of files with more content than is in the repository itself.

If a Release file is available:

1. Download it with a command such as wget https:/.... This command saves it with the same filename as the end of the URL.
   • In some cases this might be just v4.1.1.tar.gz, which is not descriptive. You may provide a name with wget -O somepackage-v4.1.1.tar.gz https://....
2. Extract files with .tar.*, .tbz2 or similar with tar xf filename.tar.gz.
   • We recommend checking the contents with tar tf filename.tar.gz to see if it’s going to extract everything to a new subdirectory, or put files in the current directory as well.
   • For .zip files, use unzip -l filename.zip to check the directory structure first.

If a Git project doesn’t have a Release file:

1. Check to see if it has “tagged” version available.
2. Download the .zip of the “tagged” version or checkout that version with a command such as: git clone https://..../project ; cd project ; git checkout v4.1.1.
   • If there are no tags at all, then this is probably very untested software and you should scrutinize it carefully before proceeding.

Part 2: Building the code

Once you have the code, the next step is to identify the build system. Look at the files at the top level of the code. Different programming languages use different tools. The following are the basic commands, though most of them require extra arguments.

Part 3: Installing the program

Before building, you should decide where you want the software to end up after installing. By default, many packages may try to install to a location your user account can’t write to.

We recommend placing the software under the PI /work folder so that you can share the software with other members of your group. Installing in /home is possible, but has a much smaller quota. Building in /project will be much slower, and running from there may also have run-time performance impacts. Since /scratch is temporary, software installed there will be lost.

Each build system has its own way of specifying the location.

• For Java, you may need to edit the .xml file or at least examine it to determine how to override its default.
• For Python, use either Conda or venv.
• For C/C++ and Fortran projects, consult the following table to specify a destination directory:

After building, run the install command:

Not all Makefile based projects provide an install target, so you may just end up with a binary somewhere under the project directory, maybe at the top level, maybe not.

Notes on certain build systems

• With cmake, you can use ccmake for an interactively view of all of the possible parameters.
  • This is sometimes necessary to turn on or off certain features, or to specify dependency location information.
• For “autobuild” style packages, there is a cascade of commands to run.
  • If there is an ./autogen.sh file, it can create a ./configure file from a ./configure.in file. However, only do this if ./configure doesn’t already exist.
  • If there is a ./configure.am but no ./autogen.sh, you can try aclocal && autoconf && automake && libtoolize --force.