---
title: "Training an Identification Model"
slug: "training-an-identification-model"
updated: 2026-06-11T14:22:59Z
published: 2026-06-11T14:22:59Z
canonical: "help.hyperscience.ai/training-an-identification-model"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://help.hyperscience.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Training an Identification Model

Hyperscience extracts data from documents and converts them into a machine-readable format. We support Structured, Semi-structured, and Additional documents. To learn how to differentiate between the document types, see [Understand Document Types](/v43/docs/understanding-document-types).

> [!NOTE]
> Semi-structured use cases
> 
> Layouts for Semi-structured documents help identify and extract data from pages that do not have a consistent structure or fixed visual templates. While the information you need to extract (e.g., identification number, address) remains the same, its location may vary and appear under different names or labels. Examples of Semi-structured documents are paystubs and invoices. They contain key pieces of information that are always present, but their placement can vary significantly across different versions of the document.
> 
> Use Semi-structured layouts in the following scenarios:
> 
> - When field positions vary
> - When tables vary in size or structure across documents

Train a or model to extract data from semi-structured documents using annotated examples in Training Data Management (TDM).

This article walks you through the full process — from preparing your data to training and evaluating your model.

> [!WARNING]
> **Using features for Semi-structured documents**
> 
> This article mentions features used in the processing of Semi-structured documents. Your access to those features depends on your license package and pricing plan.
> 
> To learn which features are available to your organization and how to add more, contact your Hyperscience representative.

## Requirements for training an ID model

- Model training is handled by the Trainer, which operates independently from the main application to prevent performance degradation during document processing. Learn about the trainer in our [Trainer](/v43/docs/what-is-the-trainer) article.
  - For optimal performance, the trainer requires a dedicated machine with **at least 64GB of RAM and 16 CPU cores.** To learn more, see [Infrastructure Requirements](/deployment/docs/infrastructure-requirements#trainer).
- Learn more about the product’s limits in [Product Limits and Guidelines](/general-information/docs/product-limits-and-guidelines).
- The system's default minimum requirement to run a model training is **100 documents**.
- To train a new Identification model, it is generally recommended to have **at least 400 training documents**.

## Step 1 — Sampling documents

Before uploading documents, make sure your Dataset reflects the documents you expect to process in production. Learn how to prepare your dataset in [Preparing training data.](/v43/docs/preparing-training-data)

## Step 2 — Build a layout, add it to a release, and assign it to a flow

Once you’ve identified the fields you want to extract, create a layout to define how those fields are captured.

Your layout determines:

- which fields are extracted
- how the model learns to identify them

### Example

Consider the following insurance claim form:

![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/example_03.png)

This is a typical semi-structured document where key information is spread across sections and may appear in different locations depending on the format.

- Identification models are designed to extract this type of information consistently, even when layouts vary.

Use the interactive walkthrough below to create your layout and configure the required fields.

[Embedded content](https://app.storylane.io/demo/8zo9rybmnuev?embed=inline)

#### Guidelines

- Use unique names for your fields or columns to avoid model training failure and simplify the Annotation process.
- Make sure to set the proper data type for each field or column you create to obtain a high-performance model.
  - Learn more about data types and how to choose them in [What is a Data Type?](/v43/docs/what-is-a-data-type) and [Choosing a Data Type](/v43/docs/choosing-a-data-type).
- Ensure your configurations are suitable for the fields and columns for extraction:
  - Check Multiple Occurrences if your fields have more than one occurrence.
  - Enable the Multiline setting if required.
  - Set **Identification Supervision** to **Always** for each field you want to guarantee a manual review for.
    - Enabling this setting will always generate Field ID tasks, regardless of the machine’s confidence. To learn more, see [Scoring Field Output Accuracy.](/v43/docs/scoring-field-output-accuracy0)
  - Set **Transcription Supervision** to **Always** if there are issues in the document that could prevent the machine from reading the field or the column. That way, the system will always send it to Manual Transcription, ensuring review from your keyers. Learn more about accuracy in [Accuracy](/v43/docs/accuracy) and [Transcription Accuracy and Automation](/v43/docs/transcription-accuracy-and-automation#h_01HDK93H5N9ABD9NKR8SPVBGPN).
    - Find more field configurations in the [Defining field metadata](/v43/docs/creating-semi-structured-layouts#h_01HDKAFD1VTHZ7ECPQFTVT1H3V:~:text=and%20exiting%20below.-,Defining%20field%20metadata,-Add%20a%20new) section of [Creating Semi-structured Layouts](/v43/docs/creating-semi-structured-layouts).

> [!NOTE]
> Layout versions
> 
> Identification training always uses the **latest live layout version,** not the latest committed draft.
> 
> Example: If the live version of a layout does not have the multi-line checkbox enabled, but the latest committed version does, the model will still train on the live version (without the checkbox).

- To learn more about flows and releases, see [Flows Overview](/v43/docs/flows-overview) and [What is a Release?](/v43/docs/what-is-a-release).

## Step 3 — Training Data Management (TDM)

Use the tools in Training Data Management to control, manage, and adjust the ground truth of your training sets for Identification and Classification models. In this section, you will learn how to upload your data using TDM. Before you start:

- Ensure you meet the requirements outlined in [Requirements for training an ID model](/v43/docs/v43-training-an-identification-model#requirements-for-training-an-id-model) section of this article.
- Make sure to keep 50-100 documents for testing purposes.
  - Note that they should be representative of the data expected in production. You’ll upload them after the model training is completed.

> [!NOTE]
> **Ground truth** is manually annotated data used to train our machine-learning models. We use a subset of this data to assess the performance of your models.

Use the interactive walkthrough below to upload documents to TDM:

[Embedded content](https://app.storylane.io/demo/nuhwmexb5mtn?embed=inline)

All uploaded documents will appear on the Training Data card.

> [!NOTE]
> Table models
> 
> If you have a table in your layout, switch to the **Table Identification** tab.

Note that the status of your documents will be **Ready to annotate**. Learn more about statuses in [Training Data Management](/v43/docs/training-data-management#h_01HBZK63SHHE5VDTG2903C3NQK:~:text=The%20training%20status%20of%20the%20document%3A).

> [!NOTE]
> Using Submission data in TDM
> 
> The **Send documents to Training Data Management** setting for Identification and Classification models allows you to control whether submission data is used for model training.
> 
> It is disabled by default and can be managed from the System Settings (**Administration** > **System** **Settings**).

## Step 4 — Analyze your data

Training Data Analysis allows you to group your training documents and receive recommendations to improve the quality of your dataset. Learn more in [Training Data Analysis](/v43/docs/training-data-analysis)

> [!NOTE]
> Running Training Data Analysis
> 
> We recommend running training data analysis once you’ve uploaded your documents. The system will create groups based on the similarity of your training documents which improves the efficiency of the annotation process.

Learn how to receive insights for improving your training data in the interactive walkthrough below:

[Embedded content](https://app.storylane.io/demo/otkjht7z8u6i?embed=inline)

> [!NOTE]
> Do **NOT** edit or upload documents while the analysis is taking place, as they’ll be excluded from the analysis.

The results will appear in the Training Data Health card. Learn more in [TDM for Identification Models](/v43/docs/v41-draft-tdm-for-identification-models).

### Analysis results

The results show you the eligibility and importance of each document. Learn more in [Training Data Management Features](/v43/docs/#h_01HBXJP7YMR3GZNDPPKVYFMP93).

> [!NOTE]
> Re-analyze your data
> 
> The system does not reanalyze the training data automatically. Make sure to re-analyze when:
> 
> - you upload new documents or
> - you edit the existing documents.

- **Groups** - Training data analysis groups your training set by visual similarity. For the best data representation, we recommend having at least 10 groups of each document type.

> [!NOTE]
> Groups with Excess Documents
> 
> Having a group with excess documents (e.g., more than 15 samples for Field ID and 20 samples for Table ID) does not necessarily mean that you need to remove the excess data. Depending on the specific use case and the performance of your model, you may want to enrich the annotations by adding more annotated examples from a particular group. Contact your Hyperscience representative for more information.

- **Importance** - The Training Data Curator labels each training document as having high or low importance.

> [!NOTE]
> Training Data Curator
> 
> The importance is calculated by determining which data would best contribute to the model’s performance. For each group of documents, the system labels the most impactful ones as having high importance. The goal is to improve the efficiency of the annotation process by requesting an optimal subset that reflects the variety of documents whose data you expect to identify with the model. Learn more about how data is curated in [Training Data Curator](/v43/docs/training-data-curator).

- **Eligibility** - with Document Eligibility Filtering, you can see which documents are incompatible with training and why, allowing you to address any issues accordingly and achieve better model performance. Learn more in [Document Eligibility Filtering](https://help.hyperscience.com/v41/docs/document-eligibility-filtering).
- **Detect anomalies** - Re-analyze your data and find inconsistencies across your annotations with Labeling Anomaly Detection. For more information, see our [Labeling Anomaly Detection](/v43/docs/labeling-anomaly-detection) article.

Consistent annotations are crucial for a high-performance locator model.

Learn how to annotate fields in [Field Identification](/v43/docs/field-identification) and how to annotate tables in [Table Identification](/v43/docs/table-identification).

## Step 5 — Annotate your documents

Learn how to annotate your documents by following the best practices listed below.

### General guidelines

1. Once you analyze the data, you’ll be able to annotate by group. Doing so provides you with more control over the dataset. Annotating by group and by priority helps you determine which groups have more documents and which groups are underrepresented.
2. After annotating 2-3 documents per group, you’ll be able to use guided data labeling. This feature gives suggestions provided by the machine that will help you to annotate more quickly.
3. Follow the general rule for annotating: left to right, top to bottom.
4. Make sure to maintain consistent annotations for your fields or columns. When a single value of a field or a column appears in different sections of the document, annotate it strictly in one location to avoid confusing the model.

> [!NOTE]
> Search by Text Segment
> 
> Search by Text Segment is available in TDM for **Field Identification** and **Table Identification** models.
> 
> This feature allows you to search for fields or cells by specific text segments directly within the Training Data Management interface. Search by Text Segment helps you locate, review, and annotate data faster during the model-training process. Learn how to use Search by Text Segment in [Field Identification](/v43/docs/field-identification). To learn more about segments, see our [Text Segmentation](/v43/docs/segmentation) article
5. Always use the machine predictions when drawing the bounding box. Avoid drawing it manually.
6. Adjust the machine predictions ONLY if the bounding boxes are overlapping and preventing the proper extraction of the data.
7. Do **NOT** interchange fields or columns, as doing so may lead to uncertainty for the model.
8. If a field or a table cell is not present, do not replace it with a similar value.
9. If you don’t see a box made of dashed lines around a value, do NOT annotate it. If there is no such box, it means that our internal ML models are not reading any values for that field or cell.

> [!NOTE]
> The annotations serve as Ground truth labels that guide the model through the training process. Aligning the annotations with the machine’s predictions will ensure that the model learns from accurate and consistent information. Inconsistencies, such as annotating the same information in different locations within a document, can affect the model’s ability to learn patterns accurately, which may result in lower performance or incorrect predictions. To learn more see our [Text Segmentation](/v43/docs/segmentation) article.

#### Field Identification

1. Annotate fields with Multiple Occurrences **only** when multiple instances of a field are present. Learn more about Multiple Occurrences in [Field Identification](/v43/docs/field-identification#h_01HC2BR86PNQ1GS3QWA60WXYZV:~:text=Table%20Identification.-,Fields%20with%20multiple%20occurrences,-The%20Multiple%20Occurrences).
2. Use multiple bounding boxes when a text is logically connected. Learn more in the [Multiple bounding boxes for fields](/v43/docs/field-identification#h_01HC2BR86QXNCNMABSH771G853) section of [Field Identification](/v43/docs/field-identification#h_01HC2BR86PADG44BRD3876F507:~:text=Field%20Identification%20Model-,Multiple%20bounding%20boxes%20for%20fields,-To%20annotate%20values).
3. If you don’t see a value for a field (i.e., the field is blank), do NOT annotate it.

#### Table Identification

1. When annotating a table, make sure to select a row where all data is present. The row you select is your template row, or the row in your table that is most representative of the table’s content.

> [!NOTE]
> The template row doesn't need to be the first row in the table. Hyperscience uses the copycat tool to populate the annotation from the template row to the rest of the rows. The copycat is not always accurate, so make sure to double-check the annotations before you submit.
2. Always find your table's first and last rows and ensure they are properly annotated.
3. Always press the **ESC** button before submitting a table to ensure the annotations are correct.
4. Draw one large bounding box capturing all rows of your table, and press the **S** button on your keyboard. That way, you’ll activate the Split tool and be able to define or correct the rows of your table faster. Make sure to double-check the annotations.

> [!NOTE]
> Tags
> 
> Starting in v43, you can manage documents by assigning tags during annotation.

Learn more tips and tricks on annotating tables in [Table Identification](/v43/docs/table-identification#h_01HKQ136XN8E1RJXR17A0J8TRW).

> [!NOTE]
> Once you’re ready with your annotations, re-analyze the data, and use Anomaly Detection to ensure that your annotations are correct and consistent. Learn more in [Labeling Anomaly Detection](/v43/docs/labeling-anomaly-detection). You can reanalyze your data after each iteration to maximize the quality of the training set.

### Next steps

- Check if all training documents are eligible for training.
  - The number next to **Eligible for training** on the model details page is the number of documents that will be used in your training set. This number may change as documents are annotated and each time you analyze your training data.
- Ensure you have the required number of training documents.
  - The number of **Required documents** shown on the model details page is the number of additional documents you need to upload and annotate to run a model training.

## Step 6 - Review your flow and train your model

Once you’ve reviewed your annotations and addressed any potential anomalies, you’re ready to initiate model training.

Before you run a model training:

### Review the flow’s configurations:

Your system might consist of several workflows, called flows. Each flow contains blocks, representing important stages of the data-extraction process. Learn more in [Flows Overview](/v43/docs/flows-overview#flows-overview-0).

For more precise control over the process, you can configure your flow’s settings.

- Set your **Target Accuracy** to achieve better performance.

> [!NOTE]
> The system uses QA data and the **Field Identification Target Accuracy** or **Table Identification Target Accuracy** values to calculate the optimal confidence threshold that will allow the system to reach the target accuracy with the minimum amount of manual effort. We recommend using the default values (95% for Field ID and 96% for Table ID) for the initial training to compare the results with the next iterations and adjust accordingly later. Change the target accuracy as follows after the first iteration:
> 
> - If you want to achieve high automation, set a lower percentage.
> - If you need high accuracy, set a higher value.
> 
> To learn more, see [Accuracy](/v43/docs/accuracy) and [Automation](/v43/docs/automation0).

### Run Training

Initiate a model training by clicking the **Run Training** button.

> [!NOTE]
> The button will be grayed out if you don’t have the minimum number of required documents.

You’ll receive a notification in the **Notification** section (![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/25457696203917.png)), which is located in the upper-right corner of the application, once the training is completed.

> [!NOTE]
> A single Trainer attached to your instance will train one model at a time. For example, if you run a model training for Field ID, and then start a model training for Table ID, the one that you’ve started first will be running, and the second one will be queued. To learn more, see our [What is the Trainer?](/v43/docs/what-is-the-trainer) article.

- Monitor the training jobs in the Running and Queued cards on the Trainer page (**Administration** > **Trainer**).

## Step 7 — Evaluate the training results

### **Deploy your model**

Once the model training is complete, you’ll find the candidate model in the model details page. To deploy it, click on your candidate model, then click **Deploy Model**.

The model is now live and ready for document processing. You can see insights on the automation and accuracy on the model details page. Learn more in [Model Validation Tasks](/v43/docs/model-validation-tasks) and [Evaluating Model Training Results](/v43/docs/evaluating-model-training-results).

> [!NOTE]
> Аlways verify layout-version compatibility when switching between model versions
> 
> The Live version of a model always uses the **most recent layout version**, regardless of which layout version it was originally trained with.
> 
> This pairing can lead to unexpected behavior, especially if changes were made to the layout after training (e.g. new fields, field-setting updates).
> 
> **Example:** If a model is trained on v3 of layout, and v4 of that layout is created after the training, the model will use v4 of the layout when deployed.

### **Evaluate the performance**

Use the documents you’ve chosen for testing purposes to evaluate the performance of your model. Note that, to measure the performance accurately, these documents should not be ones that were used for the training. To learn more about evaluating the model’s performance, see our [Monitoring Model Performance](/v43/docs/monitoring-model-performance) article.

- Before you start, ensure that your flow configurations match the ones you expect in production.

> [!NOTE]
> QA Sample Rate
> 
> The QA Sample Rate value represents the percentage of documents selected for Field ID, Table ID, or Transcription QA tasks. To learn more, see our [Accuracy](/v43/docs/accuracy) article.
> 
> Learn more about flow-level configurations in [Document Processing Subflow Settings](/v43/docs/document-processing-subflow-settings) article.
  - Enable **Manual Identification Supervision** if you have fields that you want to review manually. Doing so will generate Manual Identification tasks, which should be performed by a keyer.
  - If required, enable any combination of **Field Identification Quality Assurance**, **Table Identification Quality Assurance**, and **Transcription Quality Assurance**.
    - Set the QA sample rate for each type of quality assurance you enable (**Field Identification QA Sample Rate**, **Table Identification QA Sample Rate**, or **Transcription QA Sample Rate**).

### **Upload your testing documents**

Upload your documents as submissions by following the steps below:

1. Go to **Submissions**.
2. Click **Create Submission**.
3. Upload the testing documents. If you’re uploading multiple documents at once, select **One Submission per file** to evaluate the performance for each document.
4. Click **Next**.
5. Choose the flow you’re using for the model from the **Flow** drop-down list.
6. Choose the layout used for the model from the **Layout** drop-down list.
7. Click **Upload**.

### **Results**

Observe the results based on your flow settings on the **Document Output page**. Learn more in [Document Output Page.](/v43/docs/document-output-page)

> [!NOTE]
> Next Steps
> 
> - If the model is performing poorly, we suggest going over the training documents, as described in [Improving Model Performance](/v43/docs/improving-model-performance):
>   - Check for potential annotation errors and inconsistencies and fix them.
>   - Re-run training data analysis to use the labeling anomaly detection for more accurate results.
>   - Based on the results, you can also decide to enrich the training set by adding more documents.
> - If the model is performing well and the projected automation meets the target accuracy, we do not recommend retraining the model unless
>   - changes are made to the layout or
>   - the data distribution of incoming documents has changed (e.g., new visual template). Learn more in the section below.

## Retraining existing models

When you adjust a semi-structured layout, some of the changes require retraining your Identification Models, while others do not.

Retraining ensures that the model can correctly recognize new or updated fields and tables, and helps prevent unexpected behavior in production.

This article explains which types of layout changes require retraining and which ones can be applied without it. Understanding this distinction helps you save time and maintain model accuracy.

- To learn how to train an Identification model, see Training a Semi-structured model
- Learn how to monitor and improve your models in our [Monitoring Model Performance](/v43/docs/monitoring-model-performance) and [Improving Model Performance](/v43/docs/improving-model-performance) articles.

### When retraining is required

#### Field Identification Models

Retraining is required if:

- A **new field** is added.
- The **Multiline** setting for an existing field **is** **toggled**. Learn more in [Creating Semi-structured Layouts](/v43/docs/creating-semi-structured-layouts).
- The **Multiple Occurrences** setting is **toggled**.

> [!NOTE]
> Retraining required
> 
> Adding a new field without retraining will not make it functional.
> 
> You must retrain the model with documents that **include** **annotations** for the new or updated field. Until retrained, the new field will remain unsupported. You’ll see a message in Training Data Management when new fields are missing from the training data:
> 
> ![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/new fields.jpg)

#### Table Identification Models

Retraining is required if:

- A **new column** has been added.
- The **Multiline** setting for an existing column **is toggled.**

> [!NOTE]
> Retraining required
> 
> Adding a new column without retraining is not enough. The model must be retrained with **annotated** documents that contain the new or updated column. You’ll see a message in Training Data Management when new columns are missing from the training data:
> 
> ![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/table_columnsnew.jpg)

### When retraining is not required

Updating any of the following settings in an existing field does **not** require retraining your Field or Table ID model:

- **Output name**
- **Transcription Supervision**
- **Identification Supervision**
- **Required**
- **Not in English**

![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/settings.jpg)

> [!NOTE]
> Аlways verify layout-version compatibility when switching between model versions
> 
> The Live version of a model always uses the **most recent layout version**, regardless of which layout version it was originally trained with.
> 
> This pairing can lead to unexpected behavior, especially if changes were made to the layout after training (e.g. new fields, field-setting updates).
> 
> **Example**: If a model is trained on v3 of layout, and v4 of that layout is created after the training, the model will use v4 of the layout when deployed.

#### Field ID models

No retraining needed if:

- A field is removed
- A field’s data type is changed. To learn more, see [Data Types](/v41/docs/data-types).

#### Table ID models

A Table ID model does not need to be retrained if:

- A column is removed
- A column’s data type is changed

### Additional Considerations

If you retrain with existing training data only, new fields/columns or ones with updated Multiline setting will not be included. Retraining must be done with enough annotated examples covering the new or modified layout elements.

A tool used to annotate, manage, import, and export training documents. It is also used to train models by working directly with the training data (“ground truth”) obtained from each document in the training set.

A separate machine dedicated to handling resource-heavy tasks like training Identification models. It operates independently and connects to the main application through the API.

A group of documents used to help the system learn or improve. Datasets are used for training, testing, or evaluating how well the system reads and extracts information.

**Annotation** refers to a user-provided input that defines the correct prediction for a given machine learning task. Annotations are used to train supervised machine learning models.

A property that defines the format of the data expected in a field, like numbers, dates, or email addresses. For example, the data type Date accepts only valid dates (e.g., MM/DD/YYYY). Data types help Hyperscience understand what’s expected in a field and flag anything that doesn’t match.

Multiple Occurrences (MOs) are used to identify multiple different instances of a field.

**Multiline** is a setting in the Layout Editor for fields or columns that require more than one line of text (e.g., Address, Description). Enabling this setting helps the machine process these fields more efficiently.

A tool used to annotate, manage, import, and export training documents. It is also used to train models by working directly with the training data (“ground truth”) obtained from each document in the training set.

A feature in Training Data Management that indicates whether a document is eligible for training based on internal checks in the application and our machine learning logic. It provides additional information about documents that were excluded from the training set.

**Labeling Anomaly Detection** is a feature in TDM that allows you to identify potential discrepancies in the training datasets before running a model training.

Manually annotated data used to train our machine learning models. We use a subset of this data to assess the performance of your models