Press CTRL+K to open search

Enabling Application Machines with GPUs in On-Premise Docker Deployments

  • Updated on Oct 21, 2026
  • Published on May 6, 2026
  • 4 minute(s) read
Prev Next

This procedure sets up an application machine with a GPU to run ORCA (Optical Reasoning and Cognition Agent) VLMs in an on-premise Docker deployment of Hyperscience. It is for system administrators.

To use ORCA VLMs, your instance needs at least one application machine that has both a GPU (graphics processing unit) and a CPU (central processing unit).

Availability

Application machines with GPUs are supported in v41 and later, and only for ORCA use cases.

Prerequisites

Complete the following checks and stage 1 before you extract the Hyperscience bundle on the application machine. For the installation procedure, see Technical Installation Instructions.

Verify the GPU hardware

Make sure that your GPU meets the hardware requirements in Infrastructure Requirements.

Verify that your GPU supports CUDA

CUDA is a parallel computing platform and programming model created by NVIDIA. To check whether your GPU supports CUDA, run the following command:

lspci | grep -i nvidia

For more information, see NVIDIA’s CUDA GPUs - Compute Capability and NVIDIA CUDA Installation Guide for Linux.

Verify that you have a supported version of Linux.

Follow the instructions in NVIDIA’s NVIDIA CUDA Installation Guide for Linux to check your version of Linux. Then, make sure your Linux version is supported by the latest CUDA Toolkit by reviewing NVIDIA’s NVIDIA CUDA Toolkit Release Notes.

Make sure that your Linux distribution and version are supported by Hyperscience. Compatibility with the CUDA Toolkit alone is not enough. For the supported distributions and versions, see Infrastructure Requirements.

Verify that gcc is installed

The CUDA Toolkit requires the gcc compiler. To check that gcc is installed, follow the instructions in NVIDIA’s NVIDIA CUDA Installation Guide for Linux.

Verify the kernel headers and development packages

The CUDA driver requires the kernel headers and development packages for the kernel version that your system runs. It requires them when you install the driver and whenever the driver is rebuilt. For example, if your system runs kernel version 3.17.4-301, you need the 3.17.4-301 kernel headers and development packages.

To install them, or to confirm that they are installed, run the following command on Ubuntu:

apt-get install linux-headers-$(uname -r)

For the commands for other Linux distributions, see NVIDIA’s NVIDIA CUDA Installation Guide for Linux.

1. Install the CUDA drivers

In a Docker deployment, the Hyperscience application images include the CUDA Toolkit. The CUDA drivers on the host OS must be compatible with that toolkit version. For toolkit and driver versions, see NVIDIA’s NVIDIA CUDA Toolkit Release Notes.

CUDADriversAndToolkits.png

Remove existing CUDA drivers

Remove any CUDA drivers that are installed on the system. For example, on Ubuntu, run the following command:

apt-get clean; apt-get update; apt-get purge -y cuda*; apt-get purge -y nvidia-*; apt-get -y autoremove

Adjust the command for your Linux distribution.

Install nvidia-container-toolkit or nvidia-docker2.

The Docker host needs to be prepared before it can expose your GPU hardware to the containers. Although containers share your host’s kernel, they cannot access information on the system packages you have installed. A plain container will lack the device drivers that interface with your GPU.

(Ubuntu 21 and later) Install nvidia-container-toolkit.

You can activate support for NVIDIA GPUs by installing NVIDIA’s Docker Container Toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
        curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
          sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
          tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
        apt-get update
        apt-get install -y nvidia-container-toolkit

Then, verify the contents of your /etc/docker/daemon.json, as described below.

(Ubuntu 20) Install nvidia-docker2.

To activate support for NVIDIA GPUs, install nvidia-docker2 by running the following command:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
        && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
        && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
        apt-get update && apt-get install -o Dpkg::Options::="--force-confold"   -y nvidia-docker2

Then, verify the contents of your /etc/docker/daemon.json, as described below.

(Ubuntu 20 and later) Check the daemon file.

Make sure your Docker daemon file looks like the one shown below:

{
        "log-driver": "json-file",
        "log-opts": {
        "max-size": "10m",
        "max-file": "10"
        },
        "runtimes": {
        "nvidia": {
        "path": "/usr/bin/nvidia-container-runtime",
        "runtimeArgs": []
        }
        },
        "default-runtime": "nvidia",
        "node-generic-resources": [
        "NVIDIA-GPU=0"
        ]
}

Inspect your /etc/docker/daemon.json file to confirm that the configured container runtime has been changed. The system will handle the injection of GPU device connections when new containers start.

Install the latest CUDA drivers

Run the following command to install the latest CUDA drivers:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
dpkg -i cuda-keyring_1.0-1_all.deb
apt-get update && apt-get -y install cuda-drivers

We recommend the latest CUDA driver. For the minimum driver version that your version of Hyperscience requires, see Infrastructure Requirements.

Checkpoint: The latest CUDA drivers are installed, and /etc/docker/daemon.json sets nvidia as the default runtime.

2. Initialize ORCA on the GPU machine

Initialize ORCA on the application machine that has the GPU. The system then uses that machine for tasks that require ORCA. Run the following commands on the GPU machine.

  1. If you have not completed the standard installation steps on this machine, run them:

    ./run.sh init
./run.sh

  2. Run the ORCA-specific installation step for your ORCA base model. Run this step only on the GPU machine.

    • For ORCA 1.0:

      ./run.sh ipm VISION_LANGUAGE_MODEL_GPU

    • For any other ORCA base model, such as ORCA 2:

      ./run.sh ipm VISION_LANGUAGE_MODEL_GPU_2

Next steps

To install an ORCA base model, see Installing ORCA VLMs.

© Hyperscience