Vision Language Model Quality Assurance

This feature is available in v41.2 and later.

Accessing this feature

Your access to the feature described in this article depends on your license package and pricing plan.

To learn which features are available to your organization and how to add more, contact your Hyperscience representative.

If you're using the "Vision Language Model Flow via GPU" flow and would like to monitor the accuracy of the VLM's output, the system can generate Vision Language Model Quality Assurance (VLM QA) tasks for your keyers. When verifying the predictions of the VLM, keyers to not need to transcribe text or correct any incorrect transcriptions.

Starting VLM QA tasks

Keyers can begin working on Vision Language Model QA (FPT QA) tasks by following these steps:

1. Go to Tasks > Perform Tasks.

2. In the QA task type card, click Perform Tasks for Vision Language Model Quality Assurance.

Performing VLM QA tasks

At the beginning of a Vision Machine Language Model QA task, the keyer can review the provided task guidance. Then, for each field listed in the right-hand sidebar, they are  asked to:

1. find the field in the document, and

2. if the field is present, verify that the machine’s transcription of its value is correct.

For each field’s transcription, the keyer marks the text as correct, incorrect, or illegible by clicking one of the buttons at the bottom of the page or using one of the keyboard shortcuts provided. If a field is not present in the document, and the machine did not detect the field, then the keyer should mark the field’s prediction as correct.

The result is shown next to the field’s transcription in the right-hand sidebar ( a checkmark () for correct, an “X” () for incorrect, and a crossed-out eye () for illegible), if a transcription exists. Reviewed fields appear in gray text, and fields yet to be reviewed appear in black text.

Multiple occurrences of a field

Some fields may contain multiple occurrences in a document that need to be reviewed individually. If the machine detected multiple occurrences of a field, the number of occurrences found is shown in the segment's record in the right-hand sidebar.

While reviewing the document, a keyer may find additional occurrences of a field that the machine didn’t detect. In these situations, the keyer can click Add another in the field’s record in the right-hand sidebar. The keyer does not need to provide a transcription for the additional occurrence.

When a keyer marks any of a field’s occurrences as incorrect or illegible, the entire field is marked in the same way, regardless of whether any other occurrences of the field were marked as correct. In these cases, the system advances the keyer to the next field, even if there are occurrences of the current field that have not yet been reviewed.

Completing tasks

After the keyer reviews a field, the system advances the keyer to the next field in the list. The keyer can go to any individual field by clicking on it in the right-hand sidebar or using the Next or Back keyboard shortcuts.

When the keyer has marked each field as correct, incorrect, or illegible, they can submit the task by clicking Submit or pressing Enter or Return on their keyboard. This option is not available to the keyer until all fields have a correct, incorrect, or illegible designation.

Available actions

Keyers can take the following actions to help them complete VLM QA tasks:

• Show guidance — Clicking the question mark icon () in the upper-right corner of the page-viewing area reveals the guidance shown at the start of the task.

• Zoom in or Zoom out — Keyers can click the magnifying glass icons to zoom in on () or zoom out of () their view of the page.

• Show keyboard shortcuts — Clicking the keyboard icon () reveals the keyboard shortcuts.

• Hide task sidebar — If keyers need to view more of the page at a time, they can hide the right-hand sidebar by clicking the Hide task sidebar icon ().

Invalid tasks

If a document is unreadable, blank, or rotated incorrectly, a keyer can mark the task as invalid. To do so, they can click the menu at the top of the page ( ) and then click Invalid task.

After a task is marked as invalid, the task ends. The system does not count the task as an incorrect machine read, and it does not include it in accuracy calculations.

VLM QA keyboard shortcuts