Technical Upgrade Instructions

Prev Next

This article provides the steps necessary to upgrade a running Docker/Podman Hyperscience Application and Trainer to a newer version. It is oriented toward the system administrators who will perform this.

To install Hyperscience for the first time, see Technical installation instructions.

In the most common use case, you take the following steps on the application and trainer virtual machines:

  1. Download, distribute, and unpack the Hyperscience bundle

  2. Carry over the environment variables from the previous version

  3. Start the application and the trainer

Disable swap before you run the run.sh script

This applies to both installations and upgrades, on both application and trainer VMs. If swap is enabled, run.sh stops and returns the error "Having the swap enabled can potentially lock the system for a long time. We DO NOT recommend having the swap enabled. Refer to the installation instructions for more details about disabling it."

For the steps to turn swap off, see Disable swap space in Hyperscience Virtual Machine Prerequisites.

Before you begin

Make sure you are following the steps for the upgrade process as described in The Upgrade Process.

Make sure you are either a root user or that you have elevated root-level permissions (that is, you can execute sudo commands) on the machines that run Hyperscience.

If you are upgrading from v38 or earlier, or if your current license key is expired, request a license key from your Hyperscience representative before you begin. See License Keys.

Before proceeding with the upgrade, back up your database and file store if you have not already. This helps avoid losing data, and you can use the backup if you ever need to roll back to the previous version.

Download, distribute, and unpack the upgrade bundle

The Hyperscience Application and Trainer come bundled in a single archive file named hyperscience-trainer-x.x.x.tgz, where x.x.x is the version number.

Due to the archive's size, there is a limit on the number of times you can download a single version. If you are upgrading Hyperscience on multiple virtual machines, make sure you download it once and then distribute it internally within your own network.

To download the bundle on a Linux virtual machine, execute this command:

sudo wget <link-to-bundle>

Once the bundle has been downloaded and distributed to all virtual machines, move it to the installation folder /opt/hs (recommended) and extract it:

sudo tar -xzvf hyperscience-trainer-x.x.x.tgz

This extracts the file into a folder with the name of the archive. The folder contains a new ".env" file, which you replace with the one from your previous version.

Upgrade the application

Complete the following steps on each application virtual machine in the cluster, in the order given.

1. Check SELinux status (RHEL only)

Run the following command on the machine:

sudo sestatus
# Sample Output
# SELinux status:     enabled / disabled
# Current mode:       enforcing / permissive

SELinux operates on the principle of default denial: anything not explicitly allowed is denied. When enabled, SELinux can operate in two global modes:

  • Permissive mode, in which permission denials are logged but not enforced.

  • Enforcing mode, in which permission denials are both logged and enforced.

If SELinux is enabled in permissive mode or disabled, proceed to the next step.

If SELinux is enabled in enforcing mode, you also need to give your application containers access to the archive files that contain flow blocks. These blocks are required to process submissions in Hyperscience.

To give the necessary permissions, run the following command against the new version's folder:

chcon -t container_file_t -R <installation_folder>/sdm_blocks

If you do not, you see the following error when you attempt to initialize the application:

PermissionError: [Errno 13] Permission denied: '/var/www/forms/forms/sdm_blocks'
CommandError: Command [/var/www/post_migrate.sh] exited with 1

2. Carry over the ".env" file

You already have a ".env" file set from the previous version. Copy it from the old version folder to the new one, replacing the file that came with the bundle.

If you are upgrading to Hyperscience v38 or later and you are using a load balancer, add the HS_CSRF_TRUSTED_ORIGINS variable to your ".env" file to prevent login issues:

HS_CSRF_TRUSTED_ORIGINS=https://test.com,https://*.subdomain.com

For more information, see the Load Balancing section of Hyperscience Infrastructure Prerequisites.

Remove any ".env" variables that are left without a value to prevent startup issues.

If you have more than one application virtual machine, distribute the same ".env" file across all machines. We validate the file's checksum at startup, and failing to do so results in startup failure on subsequent virtual machines.

3. Initialize the application on the first machine

When upgrading to a new version, and every time the ".env" file is edited, you must initialize the application on the first application virtual machine in the cluster.

Go to the folder where the new Hyperscience bundle was unpacked, and run the initialization script:

sudo bash run.sh init

This script updates the database tables and prepares the application for startup. There is no need to run it on all machines in the cluster.

4. Start the application on all machines

After running the init command, update the running application containers on each machine in the cluster:

sudo bash run.sh

This automatically starts the required application roles based on the configuration you have set. As long as Docker or Podman is configured to start at boot, the application also starts at boot.

5. Enter a license key

Upgrading to v39 or later requires a valid license key. If you upgraded from v38 or earlier, your instance has no key yet, and the application prompts a System Admin for one at the first login after the upgrade. No other functionality is available until a valid key is entered. Each key is valid for a single instance, so request one from your Hyperscience representative before your upgrade window. You need to give them the browser URL of your application and the instance type (e.g., PROD, UAT or DEV) the key applies to. For the steps to enter the key, see Providing a license key during installations or upgrades in License Keys.

If your instance already has a license key that is still valid, you do not need a new one. To check the status and expiration date of your current key, go to the System & Health page (Administration > System & Health) before you start the upgrade. For more information, see License key statuses in License Keys.

Enter the license key before you upgrade the trainer

The trainer connects to the application with an authentication token that you copy from the Users page. You cannot reach that page until the application is unlocked with a valid license key.

Upgrade the trainer

The steps to upgrade the trainer are similar to upgrading the application, but the trainer uses its own local PostgreSQL database that is provided with the bundle, along with a local file store, so there is no separate database to point it at.

1. Check SELinux status (RHEL only)

Run the following command on the machine:

sudo sestatus
# Sample Output
# SELinux status:     enabled / disabled
# Current mode:       enforcing / permissive

SELinux operates on the principle of default denial: anything not explicitly allowed is denied. When enabled, SELinux can operate in two global modes:

  • Permissive mode, in which permission denials are logged but not enforced.

  • Enforcing mode, in which permission denials are both logged and enforced.

If SELinux is enabled in permissive mode or disabled, proceed to the next step.

If SELinux is enabled in enforcing mode, you also need to give your trainer containers access to the media and database directories. The database directory is specific to the version you are upgrading to, so create and label it for the new version. Run the following commands, assuming that you are using the default HS_PATH=/mnt/hs variable:

chcon -t container_file_t /mnt/hs/trainer_media
mkdir -p /mnt/hs/postgres_trainer<major_version_number>_<minor_version_number>
chcon -R -t container_file_t /mnt/hs/postgres_trainer<major_version_number>_<minor_version_number>

# Hyperscience version numbers are formatted as follows:
# <major_version_number>.<minor_version_number>.<patch_version_number>

# For example, if your application version is 42.3.2, you would enter:
# chcon -t container_file_t /mnt/hs/trainer_media
# mkdir -p /mnt/hs/postgres_trainer42_3
# chcon -R -t container_file_t /mnt/hs/postgres_trainer42_3

2. Carry over the ".env" file

You already have a ".env" file set from the previous trainer version. Copy it from the old version folder to the new one.

3. Start the trainer

First, obtain an authentication token from a user with API permissions. This user can be any of these types:

  • Dedicated local trainer user

  • User created by an external authentication method

  • Any local user with the proper permissions

The details on how to set up the first two options are described in Connect the trainer through local or external users.

Once you choose the type of user to get the token from, follow these steps to obtain the authentication token:

  1. Log in to the application.

  2. Click Users, and select the chosen user.

  3. Find the user's Authentication Token, and click Copy.

  4. Click Done.

Next, go to the folder where the new Hyperscience bundle was unpacked, and run the following command. This automatically starts the required trainer roles based on the configuration you have set. As long as Docker or Podman is configured to start at boot, the trainer also starts at boot.

sudo bash run.sh trainer <main application URL> <authentication token>

# <main application URL>:
# This is the URL of the main application.
# This must be specified without the trailing slash.
# You can use either the IP address or the hostname. If using the hostname,
# make sure it can be properly resolved.

# <authentication token>:
# This is the authentication token for a user provisioned in the main application that
# has the API Access permission enabled. We recommend creating a service account,
# or a "user" whose token will provide API access to the trainer and other services.
# For information on obtaining tokens, see Managing API Tokens.

Here are some good examples of how you can run this command:

sudo bash run.sh trainer http://environment.hyperscience.com 2e831cc1a420a7a45264aab0624aa26d9da1aa70

sudo bash run.sh trainer https://environment.hyperscience.com 2e831cc1a420a7a45264aab0624aa26d9da1aa70

sudo bash run.sh trainer http://10.0.0.5 2e831cc1a420a7a45264aab0624aa26d9da1aa70

sudo bash run.sh trainer https://10.0.0.5 2e831cc1a420a7a45264aab0624aa26d9da1aa70

Here are some examples of incorrect ways to run this command:

# Do not use a trailing slash on the URL:
bash run.sh trainer http://environment.hyperscience.com/ 2e831cc1a420a7a45264aab0624aa26d9da1aa70

# Do not keep the < and > brackets when substituting the variables for the command:
bash run.sh trainer <https://environment.hyperscience.com> <2e831cc1a420a7a45264aab0624aa26d9da1aa70>

# Always include either "http://" or "https://" at the start of your URL:
bash run.sh trainer environment.hyperscience.com 2e831cc1a420a7a45264aab0624aa26d9da1aa70

# Always use a fully qualified domain name, or the IP address of the application
# server if using only one application server:
bash run.sh trainer environment-shortname 2e831cc1a420a7a45264aab0624aa26d9da1aa70

The trainer connects to the main Hyperscience application through the API.

Whenever you change the configuration of the connection between the trainer and the main application — for example, the URL of the Hyperscience application changes, the application load balancer URL changes, a new API authentication token is used, SSL is configured, or LDAP is configured — you have to restart the trainer by rerunning the command above.

To configure the trainer to connect to the application over TLS, follow the guidelines in the TLS configuration for outbound connections section of Security.

4. Connect the trainer through local or external users

We have security measures to:

  • enforce the use of a single authentication method, and

  • periodically invalidate API tokens for API users created with external authentication methods. This automatic token invalidation is optional and disabled by default. To learn more, see External Authentication Methods and API Users.

If you are using an external authentication method, you can still connect your trainer to the application either through a local user or one obtained from the external authentication method.

Local trainer user

To connect through a local user, add the TRAINER_USER variable to the ".env" file of the application machine. Adding this variable ensures that your trainer's access to the application is not interrupted after upgrading.

  1. Add the TRAINER_USER variable to your application's ".env" file, with the username of your trainer's user as its value:

    TRAINER_USER=<username_of_trainer_user>
  2. Restart the application:

    sudo bash run.sh init
    sudo bash run.sh

Once you start the trainer with the new token, your trainer's user is automatically added to the list of exempted users.

Editing the TRAINER_USER variable creates a new user with the username you enter, if it does not already exist. It does not remove the user previously created through the TRAINER_USER variable.

Trainer user from an external authentication method

If you are connecting the trainer through a user created with your external authentication method, add the trainer's username to the TOKEN_REVALIDATION_EXEMPTED_USERS variable. Do not add the TRAINER_USER variable to your ".env" file. For more information, see External Authentication Methods and API Users.