Enabling Application Machines with GPUs in On-Premise Kubernetes Deployments

This procedure sets up an application machine with a GPU to run ORCA (Optical Reasoning and Cognition Agent) VLMs in an on-premise Kubernetes deployment of Hyperscience. It is for system administrators.

To use ORCA VLMs, your instance needs at least one application machine that has both a GPU (graphics processing unit) and a CPU (central processing unit).

Availability
Application machines with GPUs are supported in v41 and later, and only for ORCA use cases.

Prerequisites

On bare metal clusters, complete the following checks before stage 1.

Verify the GPU hardware

Make sure that your GPU meets the hardware requirements in Kubernetes Installation Overview.

Verify that your GPU supports CUDA

CUDA is a parallel computing platform and programming model created by NVIDIA. To check whether your GPU supports CUDA, run the following command:

lspci | grep -i nvidia

For more information, see NVIDIA’s CUDA GPUs - Compute Capability and NVIDIA CUDA Installation Guide for Linux.

Verify that you have a supported version of Linux

Follow the instructions in NVIDIA’s NVIDIA CUDA Installation Guide for Linux to check your version of Linux. Then, make sure your Linux version is supported by the latest CUDA Toolkit by reviewing NVIDIA’s NVIDIA CUDA Toolkit Release Notes.

Verify that `gcc` is installed

The CUDA Toolkit requires the gcc compiler. To check that gcc is installed, follow the instructions in NVIDIA’s NVIDIA CUDA Installation Guide for Linux.

Verify the kernel headers and development packages

The CUDA driver requires the kernel headers and development packages for the kernel version that your system runs. It requires them when you install the driver and whenever the driver is rebuilt. For example, if your system runs kernel version 3.17.4-301, you need the 3.17.4-301 kernel headers and development packages.

To install them, or to confirm that they are installed, run the following command on Ubuntu:

apt-get install linux-headers-$(uname -r)

For the commands for other Linux distributions, see NVIDIA’s NVIDIA CUDA Installation Guide for Linux.

1. Set up the GPU nodes

For ORCA to use the GPU, the Kubernetes nodes must have the correct drivers or AMIs, and the containers must be able to access the GPU. Follow the section for your platform.

Bare metal

The CUDA Toolkit is installed in the Hyperscience images, and the CUDA drivers on the host OS must be compatible with that toolkit version. For toolkit and driver versions, see NVIDIA’s NVIDIA CUDA Toolkit Release Notes.

Remove existing CUDA drivers

Remove any CUDA drivers that are installed on the system. For example, on Ubuntu, run the following command:

apt-get clean; apt-get update; apt-get purge -y cuda*; apt-get purge -y nvidia-*; apt-get -y autoremove

Adjust the command for your Linux distribution.

Install nvidia-container-toolkit or nvidia-docker2

The Docker host needs to be prepared before it can expose your GPU hardware to the containers. Although containers share your host’s kernel, they cannot access information on the system packages you have installed. A plain container will lack the device drivers that interface with your GPU.

(Ubuntu 21 and later) Install nvidia-container-toolkit.

You can activate support for NVIDIA GPUs by installing NVIDIA’s Docker Container Toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
       && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
       && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
             sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
             tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
       apt-get update && apt-get install -y nvidia-container-toolkit

(Ubuntu 20) Install nvidia-docker2.

To activate support for NVIDIA GPUs, install nvidia-docker2 by running the following command:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
        && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
        && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
        apt-get update && apt-get install -o Dpkg::Options::="--force-confold"   -y nvidia-docker2

Configure the container runtime

Run the following nvidia-ctk command to configure the container runtime. The command updates the /etc/docker/daemon.json file on the host so that Docker can use the NVIDIA Container Runtime.

nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

Inspect your /etc/docker/daemon.json file to confirm that the configured container runtime has been changed. The system will handle the injection of GPU device connections when new containers start.

Install the latest CUDA drivers

Run the following command to install the latest CUDA drivers:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_1.0-1_all.deb
apt-get update && apt-get -y install cuda-drivers

We recommend the latest CUDA driver. For the minimum driver version that your version of Hyperscience requires, see Kubernetes Installation Overview.

Deploy the nvidia-device-plugin DaemonSet

To deploy the DaemonSet, follow the instructions in the “Enabling GPU Support in Kubernetes” section of the NVIDIA device plugin for Kubernetes documentation in GitHub.

AWS EKS

Use the pre-built GPU AMIs that AWS provides with all the necessary drivers installed. For more information, see the “Amazon EKS optimized accelerated Amazon Linux AMIs” section of AWS’s Amazon EKS optimized Amazon Linux AMIs. For a list of pre-built GPU AMIs by Kubernetes version, see Amazon EKS AMI’s Releases list in GitHub.
Deploy nvidia-plugin-daemonset by following the instructions in the “Enabling GPU Support in Kubernetes” section of the NVIDIA device plugin for Kubernetes documentation in GitHub.

Azure AKS

To enable GPUs for Azure AKS, follow the steps in Microsoft’s Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS). Doing so sets up the GPU nodes and installs the nvidia-device-plugin DaemonSet.

Google GKE

Follow Google Cloud’s Run GPUs in GKE Standard node pools to set up GPU node pools with drivers and the DaemonSet.

Checkpoint: The GPU nodes have CUDA drivers, and the NVIDIA device plugin DaemonSet is deployed.

2. Initialize ORCA on the GPU machine

Initialize ORCA on the application machine that has the GPU. The system then uses that machine for tasks that require ORCA. Run the following commands on the GPU machine.

If you have not completed the standard installation steps on this machine, run them:
```
./run.sh init
./run.sh
```
Run the ORCA-specific installation step for your ORCA base model. Run this step only on the GPU machine.
- For ORCA 1.0:
```
./run.sh ipm VISION_LANGUAGE_MODEL_GPU
```
- For any other ORCA base model, such as ORCA 2:
```
./run.sh ipm VISION_LANGUAGE_MODEL_GPU_2
```

Next steps

To install an ORCA base model, see Installing ORCA VLMs.