---
title: "Training a Semi-structured Model"
slug: "training-a-semi-structured-model"
updated: 2025-08-18T15:44:49Z
published: 2025-08-18T15:44:49Z
canonical: "help.hyperscience.ai/training-a-semi-structured-model"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://help.hyperscience.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Training a Semi-structured Model

> [!WARNING]
> **Accessing this feature**
> 
> Your access to the feature described in this article depends on your license package and pricing plan.
> 
> To learn which features are available to your organization and how to add more, contact your Hyperscience representative.

Hyperscience extracts data from documents and converts them into a machine-readable format. We support Structured, Semi-structured, and Additional documents. To learn how to differentiate between the document types, see [Understand Document Types](/v41/docs/understanding-document-types).

> [!NOTE]
> Semi-structured use cases
> 
> Layouts for Semi-structured documents help identify and extract data from pages that do not have a consistent structure or fixed visual templates. While the information you need to extract (e.g., identification number, address) remains the same, its location may vary and appear under different names or labels. Examples of Semi-structured documents are paystubs and invoices. They contain key pieces of information that are always present, but their placement can vary significantly across different versions of the document.
> 
> Use Semi-structured layouts in the following scenarios:
> 
> - When field positions vary
> - When tables vary in size or structure across documents

In this article, you’ll learn how to build a robust Semi-structured model corresponding to your business needs by using our Training Data Management tools.

## Step 1 - Sampling Documents

### Review your documents

Having a diverse, representative training set is crucial for a high-quality Identification model. Selecting the appropriate documents for training will optimize your Semi-structured model’s performance.

- Determine the common types of documents you'll be processing, and ensure you have at least:
  - 15 documents for a Field Identification model, or
  - 20 documents for a Table Identification model.

> [!NOTE]
> To create a more generalized model that's able to handle a wide range of different documents, you need a diverse dataset. We recommend using documents with similar visual template for a robust model. Provide examples of each type of document you want to include. For example, if you are training a model to process invoices from multiple vendors, make sure to include at least 20 invoices per vendor to ensure high model performance.

- Become familiar with the edge cases (i.e., documents that are completely different from the main ones) and determine their variety. Exclude them if they are not suitable for your use case.
- Remove documents that would reduce model performance (e.g., documents containing unrelated information, highly distorted pages (pages that are noisy, skewed, pixelated ones, duplicates).
- Choose at least 50-100 documents for testing purposes. Note that these documents should be representative of the data you expect in production.

> [!NOTE]
> Reusing an existing model vs. training a new one
> 
> Adding more vendors’ documents to a model can be effective, but if the document structures vary too much, it may impact your model’s performance. If the formats and field requirements are similar, a single model can work well. Otherwise, training a separate model can help ensure more consistent results.
> 
> For example, if your model is trained on invoices where key fields like Invoice Number, Total Amount, and Due Date consistently appear in similar locations, adding vendors’ documents with the same structure is straightforward. But if documents from a new vendor introduce a visual layout where those fields appear in different locations — such as the Due Date in the footer or payment details split across sections — we recommend training a separate model to maintain accuracy. Learn more about model performance in [Monitoring Model Performance](/v41/docs/monitoring-model-performance)

### Review your fields and columns

- Your fields and columns should be representative of the information you want to extract.
- Ensure they are present in your documents to achieve a high-performance model.
- Review any interchangeable fields or columns, as this might result in poor model performance.

> [!NOTE]
> For optimal results, the data in your training set should be representative of the data expected in production.

## Step 2 - Build your layout and add it to a release

### Build your layout

Once you’ve determined the information you want to extract, you need to build your Semi-structured layout. To learn more about Semi-Structured Layouts, see [Determining Layout Type](/v41/docs/determining-layout-type).

1. Create your layout by following the steps in [Creating Semi-structured Layouts](/v41/docs/creating-semi-structured-layouts).

> [!NOTE]
> Use unique names for your fields or columns to avoid model training failure and simplify the annotation process.
  - Make sure to set the proper data type for each field or column you create to obtain a high-performance model. Learn more about data types and how to choose them in [What is a Data Type?](/v41/docs/what-is-a-data-type) and [Choosing a Data Type](/v41/docs/choosing-a-data-type).
  - Ensure your configurations are suitable for the fields and columns for extraction:
    - Check Multiple Occurrences if your fields have more than one occurrence. Learn more about the Multiple Occurrences (MO) checkbox in [Training a New Field Identification Model](/v41/docs/training-a-new-field-identification-model).
    - Enable the Multiline setting if required.
    - Set **Identification Supervision** to **Always** for each field you want to guarantee a manual review for. Enabling this setting will always generate Field ID tasks, regardless of the machine’s confidence. To learn more see [Scoring Field Output Accuracy.](/v41/docs/scoring-field-output-accuracy0)
    - Set **Transcription Supervision** to **Always** if there are issues in the document that could prevent the machine from reading the field or the column. That way, the system will always send it to Manual Transcription, ensuring review from your keyers. Learn more about accuracy in and [Transcription Accuracy and Automation](/v41/docs/transcription-accuracy-and-automation#h_01HDK93H5N9ABD9NKR8SPVBGPN). Find more field configurations in the [Defining field metadata](/v41/docs/creating-semi-structured-layouts#h_01HDKAFD1VTHZ7ECPQFTVT1H3V:~:text=and%20exiting%20below.-,Defining%20field%20metadata,-Add%20a%20new) section of [Creating Semi-structured Layouts](/v41/docs/creating-semi-structured-layouts).

> [!NOTE]
> Layout versions
> 
> Identification training always uses the **latest live layout version,** not the latest committed draft.
> 
> Example: If the live version of a layout does not have the multi-line checkbox enabled, but the latest committed version does, the model will still train on the live version (without the checkbox).

### Assign to a release

1. Add your layout to a release by following the steps described in [Adding a New Release](/v41/docs/adding-a-new-release)
2. Follow the steps in [Assigning a Release to a Flow](/v41/docs/assigning-a-release-to-a-flow#assigning-a-release-to-a-flow-0) to match your release to the flow you are using. Learn more about releases in [What is a Release?](/v41/docs/what-is-a-release)

## Step 3 - Using Training Data Management (TDM)

Use the tools in Training Data ManagementTDM to control, manage, and adjust the ground truth of your training sets for Identification and Classification models. In this section, you will learn how to upload your data using TDM. Before you start:

> [!NOTE]
> **Ground truth** is manually annotated data used to train our machine-learning models. We use a subset of this data to assess the performance of your models.

- See the training requirements in [Requirements for Training a New Model](/v41/docs/requirements-for-training-a-new-model).
- Make sure to keep 50-100 documents for testing purposes. Note that they should be representative of the data expected in production. You’ll upload them after the model training is completed.

### Upload your documents

1. Go to the Model Management page for your layout (**Library** > **Models**).
2. Click **Upload Training Documents** and upload each document as its own file.
3. Click **Upload** in the dialog box.

All uploaded documents will appear on the Training Documents card. Switch between the **Field Identification** and **Table Identification** tabs, depending on the type of model you want to train.

> [!NOTE]
> Using Submission data in TDM
> 
> In v41.2, we’ve introduced a new setting called **Send documents to Training Data Management** for Identification and Classification models. This setting allows you to control whether submission data is used for model training. It is disabled by default and can be managed from the System Settings (**Administration** > **System** **Settings**).

Note that the status of your documents will be **Ready to annotate**. Learn more about statuses in [Training Data Management](/v41/docs/training-data-management#h_01HBZK63SHHE5VDTG2903C3NQK:~:text=The%20training%20status%20of%20the%20document%3A).

## Step 4 - Analyze your data

Training Data Analysis allows you to group your training documents and receive recommendations to improve the quality of your dataset.

> [!NOTE]
> Running Training Data Analysis
> 
> We recommend running training data analysis once you’ve uploaded your documents. The system will create groups based on the similarity of your training documents which improves the efficiency of the annotation process.

Receive insights for improving your training data by clicking the **Analyze Data** button, located in the Training Data Health card.

![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/training_Data_analysis.jpg)

> [!NOTE]
> Do **NOT** edit or upload documents while the analysis is taking place, as they’ll be excluded from the analysis.

The results will appear in the Training Data Health card. Learn more in [TDM for Identification Models](/v41/docs/v41-draft-tdm-for-identification-models).

### Analysis results

The results show you the eligibility and importance of each document. Learn more in [Training Data Management Features](/v41/docs/#h_01HBXJP7YMR3GZNDPPKVYFMP93).

> [!NOTE]
> Re-analyze your data
> 
> The system does not reanalyze the training data automatically. Make sure to re-analyze when:
> 
> - you upload new documents or
> - you edit the existing documents.

- **Groups** - Training data analysis groups your training set by visual similarity. For best data representation, we recommend having at least 10 groups of each document type.

> [!NOTE]
> Groups with Excess Documents
> 
> Having a group with excess documents (e.g., more than 15 samples for Field ID and 20 samples for Table ID) does not necessarily mean that you need to remove the excess data. Depending on the specific use case and the performance of your model, you may want to enrich the annotations by adding more annotated examples from a particular group. Contact your Hyperscience representative for more information.

- **Importance** - The Training Data Curator labels each training document as having high or low importance.

> [!NOTE]
> Training Data Curator
> 
> The importance is calculated by determining which data would best contribute to the model’s performance. For each group of documents, the system labels the most impactful ones as having high importance. The goal is to improve the efficiency of the annotation process by requesting an optimal subset that reflects the variety of documents whose data you expect to identify with the model. Learn more about how data is curated in [Training Data Curator](/v41/docs/training-data-curator).

- **Eligibility** - with Document Eligibility Filtering, you can see which documents are incompatible with training and why, allowing you to address any issues accordingly and achieve better model performance. Learn more in [Document Eligibility Filtering](https://help.hyperscience.com/v41/docs/document-eligibility-filtering).
- **Detect anomalies** - Re-analyze your data and find inconsistencies across your annotations with Labeling Anomaly Detection. For more information, see our [Labeling Anomaly Detection](/v41/docs/labeling-anomaly-detection) article.

## Step 5 - Annotate your documents

Consistent annotations are crucial for a high-performance locator model.

Learn how to annotate fields in [Field Identification](/v41/docs/field-identification) and how to annotate tables in [Table Identification](/v41/docs/table-identification).

### Best practices

Learn how to annotate your documents easier and faster, by following the best practices listed below:

### General guidelines

1. Once you analyze the data, you’ll be able to annotate by group. Doing so provides you with more control over the dataset. Annotating by group and by priority helps you determine which groups have more documents and which groups are underrepresented.
2. After annotating 2-3 documents per group, you’ll be able to use guided data labeling. This feature gives suggestions provided by the machine that will help you to annotate more quickly.
3. Follow the general rule for annotating: left to right, top to bottom.
4. Make sure to maintain consistent annotations for your fields or columns. When a single value of a field or a column appears in different sections of the document, annotate it strictly in one location to avoid confusing the model.

> [!NOTE]
> Search by Text Segment
> 
> **Availability:**
> 
> - **v41.1** and later — Available in TDM for **Field Identification Models**.
> - **v41.2** and later — Available in TDM for **Table Identification Models**.
> 
> This feature allows you to search for fields or cells by specific text segments directly within the Training Data Management interface. Search by Text Segment helps you locate, review, and annotate data faster during the model-training process. Learn how to use Search by Text Segment in [Field Identification](/v41/docs/field-identification). To learn more about segments, see our [Text Segmentation](/v41/docs/segmentation) article
5. Always use the machine predictions when drawing the bounding box. Avoid drawing it manually.
6. Adjust the machine predictions ONLY if the bounding boxes are overlapping and preventing the proper extraction of the data.
7. Do **NOT** interchange fields or columns, as doing so may lead to uncertainty for the model.
8. If a field or a table cell is not present, do not replace it with a similar value.
9. If you don’t see a box made of dashed lines around a value, do NOT annotate it. If there is no such box, it means that our internal ML models are not reading any values for that field or cell.

> [!NOTE]
> The annotations serve as Ground truth labels that guide the model through the training process. Aligning the annotations with the machine’s predictions will ensure that the model learns from accurate and consistent information. Inconsistencies, such as annotating the same information in different locations within a document, can affect the model’s ability to learn patterns accurately, which may result in lower performance or incorrect predictions. To learn more see our [Text Segmentation](/v41/docs/segmentation) article.

#### Field Identification

1. Annotate fields with Multiple Occurrences **only** when multiple instances of a field are present. Learn more about Multiple Occurrences in [Field Identification](/v41/docs/field-identification#h_01HC2BR86PNQ1GS3QWA60WXYZV:~:text=Table%20Identification.-,Fields%20with%20multiple%20occurrences,-The%20Multiple%20Occurrences).
2. Use multiple bounding boxes when a text is logically connected. Learn more in the [Multiple bounding boxes for fields](/v41/docs/field-identification#h_01HC2BR86QXNCNMABSH771G853) section of [Field Identification](/v41/docs/field-identification#h_01HC2BR86PADG44BRD3876F507:~:text=Field%20Identification%20Model-,Multiple%20bounding%20boxes%20for%20fields,-To%20annotate%20values).
3. If you don’t see a value for a field (i.e., the field is blank), do NOT annotate it.

#### Table Identification

1. When annotating a table, make sure to select a row where all data is present. The row you select is your template row, or the row in your table that is most representative of the table’s content.

> [!NOTE]
> The template row doesn't need to be the first row in the table. Hyperscience uses the copycat tool to populate the annotation from the template row to the rest of the rows. The copycat is not always accurate, so make sure to double-check the annotations before you submit.
2. Always find your table's first and last rows and ensure they are properly annotated.
3. Always press the **ESC** button before submitting a table to ensure the annotations are correct.
4. Draw one large bounding box capturing all rows of your table, and press the **S** button on your keyboard. That way, you’ll activate the Split tool and be able to define or correct the rows of your table faster. Make sure to double-check the annotations.

Learn more tips and tricks on annotating tables in the **Table ID Supervision** tab of [Table Identification](/v41/docs/table-identification#h_01HKQ136XN8E1RJXR17A0J8TRW).

> [!NOTE]
> Once you’re ready with your annotations, re-analyze the data, and use Anomaly Detection to ensure that your annotations are correct and consistent. Learn more in [Labeling Anomaly Detection](/v41/docs/labeling-anomaly-detection). You can reanalyze your data after each iteration to maximize the quality of the training set.

### Next steps

- Check if all training documents are eligible for training.
  - The number next to **Eligible for training** on the model details page is the number of documents that will be used in your training set. This number may change as documents are annotated and each time you analyze your training data.
- Ensure you have the required number of training documents.
  - The number of **Required documents** shown on the model details page is the number of additional documents you need to upload and annotate to run a model training.

## Step 6 - Review your flow’s settings and train your model

Once you’ve reviewed your annotations and addressed any potential anomalies, you’re ready to initiate model training.

Before you run a model training:

### Review the flow’s configurations:

Your system might consist of several workflows, called flows. Each flow contains blocks, representing important stages of the data-extraction process. Learn more in [Flows Overview](/v41/docs/flows-overview#flows-overview-0).

For more precise control over the process, you can configure your flow’s settings.

- Set your **Target Accuracy** to achieve better performance.

> [!NOTE]
> The system uses QA data and the **Field Identification Target Accuracy** or **Table Identification Target Accuracy** values to calculate the optimal confidence threshold that will allow the system to reach the target accuracy with the minimum amount of manual effort. We recommend using the default values (95% for Field ID and 96% for Table ID) for the initial training to compare the results with the next iterations and adjust accordingly later. Change the target accuracy as follows after the first iteration:
> 
> - If you want to achieve high automation, set a lower percentage.
> - If you need high accuracy, set a higher value.
> 
> To learn more, see [Accuracy](/v41/docs/accuracy) and [Automation](/v41/docs/automation0).

### Run Training

Initiate a model training by clicking the **Run Training** button.

> [!NOTE]
> The button will be grayed out if you don’t have the minimum number of required documents.

You’ll receive a notification in the **Notification** section (![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/25457696203917.png)), which is located in the upper-right corner of the application, once the training is completed.

> [!NOTE]
> A single Trainer attached to your instance will train one model at a time. For example, if you run a model training for Field ID, and then start a model training for Table ID, the one that you’ve started first will be running, and the second one will be queued. To learn more, see our [What is the Trainer?](/v41/docs/what-is-the-trainer) article.

- Monitor the training jobs in the Running and Queued cards on the Trainer page (**Administration** > **Trainer**).

![](https://cdn.us.document360.io/87894cef-4958-4f3f-be6f-b75a78c82548/Images/Documentation/25457739142285.png)

## Step 7 - Evaluate the training results

**Deploy your model**

Once the model training is complete, you’ll find the candidate model in the model details page. To deploy it, click on your candidate model, then click **Deploy Model**.

The model is now live and ready for document processing. You can see insights on the automation and accuracy on the model details page. Learn more in [Model Validation Tasks](/v41/docs/model-validation-tasks) and [Evaluating Model Training Results](/v41/docs/evaluating-model-training-results).

> [!NOTE]
> Аlways verify layout-version compatibility when switching between model versions
> 
> The Live version of a model always uses the **most recent layout version**, regardless of which layout version it was originally trained with.
> 
> This pairing can lead to unexpected behavior, especially if changes were made to the layout after training (e.g. new fields, field-setting updates).
> 
> **Example:** If a model is trained on v3 of layout, and v4 of that layout is created after the training, the model will use v4 of the layout when deployed.

**Evaluate the performance**

Use the documents you’ve chosen for testing purposes to evaluate the performance of your model. Note that, to measure the performance accurately, these documents should not be ones that were used for the training. To learn more about evaluating the model’s performance, see our [Monitoring Model Performance](/v41/docs/monitoring-model-performance) article.

- Before you start, ensure that your flow configurations match the ones you expect in production.

> [!NOTE]
> QA Sample Rate
> 
> The QA Sample Rate value represents the percentage of documents selected for Field ID, Table ID, or Transcription QA tasks. To learn more, see our [Accuracy](/v41/docs/accuracy) article.
> 
> Learn more about flow-level configurations in [Document Processing Subflow Settings](/v41/docs/document-processing-subflow-settings) article.
  - Enable **Manual Identification Supervision** if you have fields that you want to review manually. Doing so will generate Manual Identification tasks, which should be performed by a keyer.
  - If required, enable any combination of **Field Identification Quality Assurance**, **Table Identification Quality Assurance**, and **Transcription Quality Assurance**.
    - Set the QA sample rate for each type of quality assurance you enable (**Field Identification QA Sample Rate**, **Table Identification QA Sample Rate**, or **Transcription QA Sample Rate**).

**Upload your testing documents**

Upload your documents as submissions by following the steps below:

1. Go to **Submissions**.
2. Click **Create Submission**.
3. Upload the testing documents. If you’re uploading multiple documents at once, select **One Submission per file** to evaluate the performance for each document.
4. Click **Next**.
5. Choose the flow you’re using for the model from the **Flow** drop-down list.
6. Choose the layout used for the model from the **Layout** drop-down list.
7. Click **Upload**.

**Results**

Observe the results based on your flow settings on the **Document Output page**. Learn more in [Document Output Page.](/v41/docs/document-output-page)

> [!NOTE]
> Next Steps
> 
> - If the model is performing poorly, we suggest going over the training documents, as described in [Improving Model Performance](/v41/docs/improving-model-performance):
>   - Check for potential annotation errors and inconsistencies and fix them.
>   - Re-run training data analysis to use the labeling anomaly detection for more accurate results.
>   - Based on the results, you can also decide to enrich the training set by adding more documents.
> - If the model is performing well and the projected automation meets the target accuracy, we do not recommend retraining the model unless
>   - changes are made to the layout or
>   - the data distribution of incoming documents has changed (e.g., new visual template). Learn more in [Retraining Existing Models](/v41/docs/retraining-existing-models).

A tool used to annotate, manage, import, and export training documents. It is also used to train models by working directly with the training data (“ground truth”) obtained from each document in the training set.

A property that defines the format of the data expected in a field, like numbers, dates, or email addresses. For example, the data type Date accepts only valid dates (e.g., MM/DD/YYYY). Data types help Hyperscience understand what’s expected in a field and flag anything that doesn’t match.

Multiple Occurrences (MOs) are used to identify multiple different instances of a field.

**Multiline** is a setting in the Layout Editor for fields or columns that require more than one line of text (e.g., Address, Description). Enabling this setting helps the machine process these fields more efficiently.

A feature in Training Data Management that indicates whether a document is eligible for training based on internal checks in the application and our machine learning logic. It provides additional information about documents that were excluded from the training set.

**Labeling Anomaly Detection** is a feature in TDM that allows you to identify potential discrepancies in the training datasets before running a model training.

Manually annotated data used to train our machine learning models. We use a subset of this data to assess the performance of your models

A separate machine dedicated to handling resource-heavy tasks like training Identification models. It operates independently and connects to the main application through the API.