Hyperscience regularly updates the Hypercell with new features, enhancements, and fixes. Major releases are created twice each year, with minor and patch versions released more frequently. To take advantage of the improvements included in each version, you need to upgrade your application.
This article is the first in a series of articles that explain concepts and considerations that should be taken into account when planning your upgrade to a new major version of Hypercell. They also walk you through the upgrade process and outline the options available to you, based on how many versions separate your current and target versions and whether you will need to continue processing submissions during the upgrade.
In this article, you’ll learn more about the compatibility of system components and artifacts across versions, as well as the core requirements and best practices to consider when planning your upgrade. More information about these concepts can be found in Planning Your Upgrade and The Upgrade Process.
Personas
The team members involved in upgrades may fall under the personas listed below:
Product Owner — the person who understands the business process to automate and where the Hypercell fits into it. They’re responsible for end-to-end testing / verification of the upgrade.
IT / System Admin — the person who manages the infrastructure that powers the Hypercell and is able to run the commands needed to upgrade, as well as run any processes before and after (e.g., backing up databases, meeting additional hardware requirements needed by a newer Hypercell version, etc.).
Annotator and Model Trainer — the person or team who maintains ground-truth data and trains models when needed
May be a person or team at Hyperscience
Flow Developer — the person or team who develops or maintains Hypercell flows
May be a person or team at Hyperscience
Keyers — the team who performs Supervision tasks to classify, identify, and transcribe data from documents
Knowledge Workers — the team who performs Supervision tasks and makes decisions in Custom Supervision tasks
Procurement — the team who is responsible for approving any additional hardware resources that may be required by newer Hypercell versions
Upgrade process overview
At a high level, you can expect your upgrade process to consist of the following steps:
Upgrade the lower environment to the target version (or an intermediate version, if needed).
Prepare your artifacts in the lower environment.
(Best practice) Test your flow and its models with a “test case” of documents to capture baseline performance. We recommend testing all of the flow’s branches to prevent halts from occurring when the flow is used in production.
Upgrade the production environment to the target or intermediate version.
Import artifacts from the lower environment to the production environment.
Test the production environment by processing submissions through the new flow.
In production, redirect incoming submissions to the imported flow.
If necessary, repeat this process until reaching the target version.
Use of multiple environments
In the world of enterprise software, updates are typically executed in a development environment, followed by the user-acceptance testing (UAT) environment, and, finally, in the production environment. Throughout this process, you should test all of the elements that you intend to use in the development and UAT phases before proceeding to production. After you upgrade, you will not be able to revert back to a previous version.
The articles in this series assume that you have at least one lower environment and a production environment. It also assumes that model training and flow creation take place in a lower environment and that models and flows are imported from that environment to production during the upgrade process. If you do not have access to a lower environment, you will need to adjust your upgrade process accordingly.
Compatibility across versions
When planning your upgrade, you should be aware that some aspects of your current Hyperscience configuration may not be compatible with your target upgrade version. Bringing them into compatibility with your target version may add to the time required to upgrade your application.
The compatibility of instance components across versions is outlined in the table below.
Component | Compatibility |
|---|---|
Trainer | Use the version that was bundled with your application version |
Layouts and releases | Can be used in the version they were created in and in all subsequent versions |
Flows | Can be used in the version they were created in (X), plus the two immediately after it (X+1 and X+2) |
Models |
|
Submissions | Can be processed in the version they were created in (X), plus the two immediately after it (X+1 and X+2) |
To learn more about compatibility across versions, see the sections below.
Trainer
Trainers aren’t upgraded; they’re overwritten. For ease of use, you can install the new trainer version on the same machine where the previous trainer version is installed.
To learn more about installing the trainer, see Technical Installation / Upgrade Instructions.
For optimal performance, we do not recommend installing a trainer on the machine where the Hyperscience application is installed.
Layouts and releases
Layouts and releases are compatible with all future versions. After upgrading the application, all layouts and releases you had on your previous version continue to function as intended.
After upgrading, layouts and releases preserve their respective statuses:
Layout statuses:
Live
Not Live
Release statuses:
Live
Locked
Draft
Flows
Flows can be used in the application version they were created in, the version directly after it, and the version directly after that version. The use of flows in any other subsequent application versions is not supported. For example, flows created in v40 can be used in v41 and v42. Hyperscience will not support the use of those flows in v43, v44, etc.
Models
Model compatibility across versions differs by model type:
Field Identification, Table Identification, and Classification models — You can use any model with any flow, as long as they were both created in the current version of the application, the previous version, or the version preceding that version.
Transcription and Text Classification models — The versions of these models must match their flow versions.
Training models
Depending on which version your models were created in and which version you’re upgrading to, you may need to retrain your models during the upgrade process.
We recommend allowing the new trainer to run any necessary training jobs for at least 2-3 days. You may benefit from allowing even more time for training, depending on the number of eligible training documents available. Only submissions that have not yet had their PII deleted can be used to train the models. Contact your Hyperscience representative to determine how long to train the models on the new trainer before upgrading the application.
Unless otherwise noted, you do not need to retrain models when upgrading to a patch version (e.g., v42.0.4 to v42.0.5) or a minor version (when available) (e.g., v42.0.x to v42.2.x).
Submissions
Because submissions are ingested or uploaded into flows, their version compatibility is the same as those flows. Therefore, assuming your flows and models are created on the current version of the application, submissions that are created before an upgrade are compatible with the next two versions. If you cannot complete them before upgrading or in one of the two versions following the version they were created in, you need to either delete them before upgrading or resubmit them as new submissions after upgrading. If any submissions are incomplete in an incompatible version, they may fail after the upgrade and have a Halted status.
For example, submissions that are created in a v40 flow are compatible with v41 and v42. If you are upgrading to v42 and still have submissions that were created in v39 or earlier, you have to resubmit these submissions as new submissions, complete (a.k.a. “drain”) them, or delete them before upgrading.
Upgrade requirements
In addition to ensuring the compatibility of your flows, models, and submissions with your target version, you will need to complete the following tasks as part of the upgrade process.
All QA tasks that are incompatible with your target version are completed or deleted.
Just as all Supervision tasks should be completed before upgrading, all QA tasks should also be completed in the version they were created in or in one of the two versions immediately following it. If it’s not possible to complete them, then they should be deleted.
Any outstanding QA tasks created more than two versions prior to your target version will remain in the queue alongside more recently created tasks, but keyers may not be able to complete them.
Upgrade sequentially.
When upgrading the application across multiple versions (e.g., v38 to v42), you cannot skip versions. For example, if you are currently running V38 and would like to upgrade to V42, you should first upgrade from V38 to V39, then to v40, then to V41, and then to v42.
Even if you don’t want to use a given application version that’s between your current and target version, you still need to upgrade to it before upgrading to your target version. Doing so allows our post-migration steps to run, ensuring compatibility and performance across each version of the application.
Best practices
If at all possible, we recommend following these best practices during the upgrade process.
Back up your environment before upgrading.
To be able to potentially revert an upgrade and prevent data loss, we recommend you do the following before starting the upgrade process:
Create a backup of your database while your application is shut down.
Create a backup of your object store while your application is shut down. You can do so while the database is being backed up.
If you need to restore previously created backups:
Clear out images and containers and start older version containers.
Restore from the desired database backup while your application is shut down.
Restore from the same point-in-time object store while your application is shut down. You can do so while the database is being restored.
Test any changes in a lower environment before upgrading in production.
We recommend testing any changes that you intend to use in production, as well as the end-to-end processing of submissions. This testing may include:
Deploying an updated flow
Testing any updated layouts
Deploying newly trained models
Deploying any updated releases
Testing any capabilities introduced in a newer version of the application
Comparing projected accuracy between models in older and newer versions