After identifying a drop in the model’s performance, the next step is to determine its root cause and address it accordingly. This article explains how to use Training Data Management (TDM) to investigate performance issues and improve your model through retraining. To detect performance reductions, follow the steps described in Monitoring Model Performance.
The approach described in this article is designed for cases where the issue is related to new or underrepresented documents with different layout patterns (e.g., different field positions), but the logic behind the extraction and annotation remains consistent.
Extraction logic and annotation rules
For training to be effective, both extraction logic (e.g., which field is captured and how) and annotation rules (how data is labeled) need to remain logically consistent, even if the document pattern varies.
Learn more about the annotation process in Text Segmentation and Training a Semi-structured Model.
When to use this approach
Use this approach when:
You’ve identified a performance drop through reporting (e.g., reduced machine accuracy or increased manual supervision, as described in Monitoring Model Performance).
Documents with new layout patterns are submitted to production.
You should not use this approach for use cases involving different extraction rules across documents. If your business logic varies by document type or layout, contact your Hyperscience representative to discuss alternative solutions.
Step 1 - Select and analyze low-performance documents
a. Select representative documents
Select documents from submissions that show low prediction quality, increased Supervision, or repeated QA corrections. These documents often have new or unseen layout patterns that may not be well represented in your training set. We recommend selecting at least 20 samples of each type of document with low performance. To learn how to download documents from the system, see the Document Renderer section in Flow Blocks.
b. Run Training Data Analysis
Upload the samples in Training Data Management (TDM), and use Training Data Analysis to assess whether the documents contributing to performance issues follow a consistent pattern or introduce too many variations for a single model to handle.
Grouping logic
Training Data Analysis groups documents based on text and location. It does not explain why a document underperforms, but it helps you assess whether retraining with more examples may improve model performance.
Note that new groups will appear every time you run the Training Data Analysis. To learn more, see Step 4 of our Training a Semi-structured Model article.
After running the analysis, review how the new documents are distributed across the groups. This distribution can help you decide whether to annotate some of them and whether more samples are needed to improve model performance.
c. Review the analysis results
Knowing the number, size, and content of the document groups can help you understand why your model performs the way it does.
Several large groups suggest consistency in the production data. Adding more annotated examples to the training set and then retraining is likely to help improve performance.
Many small groups indicate high document diversity or documents that differ in patterns and annotations. Having many small groups may lead to lower automation.
The same type of document across multiple groups suggests different layout patterns. Revisit those documents and ensure they follow the same annotation logic. Avoid introducing ambiguity in the annotation rules.
As a general rule, we recommend annotating the newly added documents.
If they belong to an existing large group, no additional action is required.
If they form a new group, aim to annotate more documents to reach at least 15–20 examples. Annotating this set of documents ensures that the group contributes effectively to model training.
Prioritizing groups
To maximize model performance, we recommend a balanced approach:
Focus on larger groups (e.g., 15+ documents) to support automation.
Add examples from smaller groups to improve model diversity and broader coverage.
This strategy helps the model generalize better while still performing reliably on high-volume formats.
Step 2 - Review annotations
After confirming that the layout pattern of your documents is consistent, the next step is to review the quality of the annotations in your training set. Even when documents are well represented, inconsistent annotations can lead to performance drops. In this step, you’ll evaluate whether the model was trained with complete and representative labels.
a. Review annotations by group
Go through the groups you’ve selected and review how the target fields are annotated.
Look for documents where these fields are missing, formatted, or annotated differently.
For example, the field “Date” may be annotated on the top left of one document and then on the top right of another document.
Importance
Use importance to decide which documents to annotate first. Importance helps prioritize examples that are expected to have the greatest impact on model training. Once a document is annotated, its importance score no longer matters. All annotated documents are used during training, regardless of their original importance level.To learn more about importance, see Training Data Curator.
b. Correct annotations
If you find inconsistencies in the annotations of your training set:
Annotate missing fields.
Correct existing annotations if needed.
Make sure the annotation rules are applied consistently across groups.
Exclude documents from training if they are not represented consistently across the dataset.
Using Labeling Anomaly Detection
Labeling Anomaly Detection identifies and highlights potential anomalies in the annotations. Learn more in Labeling Anomaly Detection.
c. Confirm coverage and re-analyze your training set
Make sure that each field is consistently represented in at least 15-20 documents. This representation ensures the model has enough examples to learn from. After making changes to the training set, re-analyze your data to get up-to-date information.
Step 3 - Retrain and evaluate the model
Once you’ve improved your training set by correcting annotations and adding representative examples, the next step is to retrain the model. This retraining generates a new model version that reflects the updated training data.
a. Initiate model retraining
Make sure your training set now includes consistently annotated, well-represented documents that reflect the observed performance issues.
In TDM, click Actions > Train new model… to start training a new version of your model.
We recommend training from scratch to reflect the updates described in Step 2. Learn more about the available training options in Incremental Training.
b. Test the new model version
After retraining, evaluate the model’s performance by using it to process a new set of 15-20 documents that match the structure of the original problematic ones. These documents should:
Follow the same pattern as the affected documents.
Not be included in the training set.
Processing documents that the model hasn’t seen helps validate whether retraining improved the model’s performance.
c. Interpret the results
If the model shows better accuracy and automation, based on the steps described in Monitoring Model Performance, you can:
Promote the model to production.
Continue monitoring performance over time for future shifts.
If performance doesn’t improve:
Recheck annotations for inconsistencies.
Confirm you’ve added enough diverse, high-quality examples.
Contact your Hyperscience representative for support in handling more complex scenarios.