NCCS On-Premise Containers

Containers In A Nutshell

Containers are small versions of operating systems that are meant to speed up the process of shipping new versions of software. As part of CI/CD, an application can be compiled and run from a container in any stage of the Software Development Life Cycle for testing or production. As part of DevOps, containers allow greater flexibility for the deployment of applications on immutable runtime systems, providing full reproducibility. As part of data science, containers allow a means of sharing code and artifacts for research reproducibility. The main idea of containers is the ability of running on a virtualized layer that can be executed from many hardware resources without any significant modifications, enabling users to have full control of their environment. This means that you do not have to ask your cluster admin to install anything for you — you can put it in a container and run.

Containers At the NCCS

The NCCS now provides the ability to run containers on the Discover, ADAPT, and Prism high-performance computing (HPC) systems. At this time, Singularity and Charliecloud are the only container runtimes available at the NCCS, with ongoing research on supporting other container runtimes. There are three main ways of running container images, these being:

Pulling base images from a Trusted Container Registry,
Building your own custom images, and/or
Transferring the Container Binary to an NCCS system.

The NCCS currently provides ways via NCCS GitLab of building both Docker and Singularity containers via CI/CD (Continuous Integration/Continuous Delivery) processes in order to test the reliability of the images before deployment, and to make them available for your team or other users to download. It is important to know that any Docker container can be converted to Singularity format when being pulled to the local systems at the NCCS. Thus, there is no need for building new images if the users current workflow leverages Docker or other Open Container Initiative format.

Using Images From Trusted Container Registries

One of the great advantages of containers is the sheer number of publicly available images for all kinds of software. The availability of these images allow users to get started quickly, but can negatively affect both the creation and runtime of the container. Some examples of these are: not being able to control what software the public image includes/removes, multiple dependency layers, not being able to control vulnerabilities in production environments, and end of support from the public images. The NCCS has adopted a hybrid system where base images are taken from the public registry and are modified through continuous integration to comply with Agency standards. These images are available in NCCS’s internal container registry, together with a set of trusted Container Registries outlined below.

NCCS Container Registry: catalog of Agency compliant application images for building custom containers and user-interactive workstations.

NVIDIA NGC Container Registry: catalog of AI/ML and HPC related containers provided by NVIDIA.

All NCCS-approved images are based on the following operating systems:

Rocky8
Rocky 9
Ubuntu18 Bionic
Ubuntu20 Bionic
Debian9
Debian10
Alpine 3

All containers have the necessary NASA baselines applied to them and also have the NASA certs loaded into their certificate store. This makes it easier for containers to interact with GitLab or other NASA tools used within the container. In order to allow for more streamlined updates, all images use repositories available to the operating system. This approach means less compiling from source and quicker image builds. In the NCCS Containers repository, there are a vast number of images to use, from the base flavors to GoLang to PostgreSQL. These ensure that the user has the necessary NASA-specific security controls applied and provide a better level of security to the container, while maintaining the stability of using images that have gone through extensive regression testing for operational validation.

Base images are available through the format listed below. These images can be pulled directly into your environment, or be used as base images to start building any application inside a container.

Examples

gitlab.nccs.nasa.gov:5050/nccs-ci/nccs-containers/base/rocky9
gitlab.nccs.nasa.gov:5050/nccs-ci/nccs-containers/base/debian10

Have an image you would like to see in the NCCS Container Registry? Submit an issue in NCCS Containers repository, create a custom one and submit a merge request, or submit a ticket to NCCS Support at support@nccs.nasa.gov. An operating system and applications support matrix is included in the NCCS Containers repository.

Tools

Singularity

Singularity is a container platform created to run complex applications on HPC clusters in a simple, portable, and reproducible way. Singularity containers are executed as a single binary file based container image, and are easy to move using existing data mobility paradigms and with no root owned daemon processes. To use Singularity on ADAPT, submit a ticket to support@nccs.nasa.gov with a list of VMs where Singularity is required. Singularity is already available by default on Discover, ADAPT, and the Prism GPU Cluster by loading the singularity module with:

$ module load singularity

For example, to pull the CentOS7 GCC8 image from the NCCS Container Registry:

$ singularity pull docker://gitlab.nccs.nasa.gov:5050/nccs-ci/nccs-containers/gcc/nccs-centos7-gcc8:latest

Another example would be pulling NVIDIA’s HPC SDK container from the NGC Container Registry:

$ singularity pull docker://nvcr.io/nvidia/nvhpc:21.2-devel-cuda_multi-ubuntu20.04

Singularity stores cache files under /home/$user/.singularity to speed up containers download processes. Due to limited storage on /home/, it is preferred that Singularity cache points to the user $NOBACKUP space. For this, setup the following two variables in your environment and/or shell source files:

export SINGULARITY_CACHEDIR="$NOBACKUP/.singularity"
export SINGULARITY_TMPDIR="$NOBACKUP/.singularity"

There are several ways of interacting with Singularity containers. The most common is getting shell inside the container:

$ singularity shell $container_name

You may also require additional local filesystem paths to be available from inside the container, for which the “-B” option is utilized. In the following example we include the $NOBACKUP space of user johndoe inside the container:

$ singularity shell -B $NOBACKUP:$NOBACKUP $container_name

For more in-detail tutorials, feel free to access the NCCS Containers repository.

By default, Singularity binary containers that are run at the NCCS will be converted to sandbox format on the fly. In order to streamline the conversion, you may want to convert the container to a sandbox from the beginning. To do so run the following:

$ singularity build --sandbox mycontainer mycontainer.sif

This will create a ‘mycontainer’ directory, from which the container can be executed as usual.

$ singularity shell mycontainer

Building Custom Images at the NCCS

Building containers from configuration files require elevated privileges. Thus, the NCCS offers three ways of building containers that can be ran locally at the NCCS.

Sylabs.io Remote Cloud – Singularity (Preferred)

Sylabs.io provides remote building capabilities that can be leveraged from inside the NCCS. Using the Remote Builder, you can easily and securely create containers for your applications without special privileges or set up in your local environment. The Remote Builder can securely build a container for you from a definition file entered here or via the Singularity CLI.

NCCS GitLab CI – Docker and Singularity (Ideal for Sharing Containers)

The NCCS has enabled a set of Continuous Integration systems that allow users to build containers in unprivileged environments and upload the container images to NCCS Container Registry. To enable the Custom Build and Container Registry features, submit a ticket to support@nccs.nasa.gov with the link to your NCCS GitLab Repository. If you do not have a GitLab account, visit the following GitLab Instructional for requesting access to the service.

The creation of custom images is highly encouraged. There are four build servers dedicated to building container images. Once a Container Registry is configured for your project, you must create a .gitlab-ci.yml file that points to the location of the container definition files. A template of this .gitlab-ci.yml is available in the NCCS Containers repository, where you will only need to remove the existing entries and add your own with either one of the following two tags to tell the GitLab Runner to use one of the build servers: “build”, “deploy”. An instructional will be available in a later date documenting this process.

How to use NCCS GitLab CI

TBD

Locally