Automation of trained models is one of the metrics that can help you determine the success of your use case.
In this article, you will learn more about automation and its dependencies.
What is automation?
Automation (also known as Skip Rate) refers to the extent to which a machine can process data independently, without requiring human supervision. It represents the proportion of extracted data with confidence scores exceeding a specified threshold. This threshold is determined by the level of accuracy you want your extracted data to have.
Automation represents the percentage of the work done by the machine for:
Classification models.
Identification models,
VLM extraction models (ORCA VLMs), and
Transcription models.
Types of automation
Projected automation
Projected automation is the predicted automation based on the desired target accuracy. It's calculated from a portion of the training data used for the model training process. This metric helps estimate how much automation can be achieved while aiming for specific performance goals. Learn more about model training in Training a Semi-structured Model.
Actual automation
The actual automation represents the real automation achieved through processing the data in your specific use case. Learn more about your model’s performance in Evaluating Model Training Results.
Automation for ORCA VLMs
ORCA VLMs availability
Starting with v42.3, accuracy reporting is available for ORCA VLMs. Learn how to enable it in Installing ORCA VLMs.
For ORCA VLMs, automation represents the percentage of fields that are processed without human supervision, based entirely on transcription confidence.
Unlike Identification models, ORCA does not separate identification from transcription. As a result:
Automation is determined solely by the confidence of the extracted value.
Fields with confidence scores above the defined threshold are processed automatically.
Fields below the threshold are routed to human supervision. To learn more, see Flexible Extraction.
ORCA automation is calculated at the field level, not the document level.
It reflects how many extracted values the system can confidently process end-to-end without human intervention.
Automation for ORCA VLMs is directly influenced by:
Confidence thresholds defined in the flow
QA sampling rates
Increasing the confidence threshold improves accuracy but reduces automation, as more fields are sent to human review. To learn more, see Vision Language Model Quality Assurance.
Accuracy and automation
There is a dependency between accuracy and automation. When your model has increased automation, more fields, including some with lower confidence, are transcribed without human oversight. On the other hand, when higher accuracy is needed, automation decreases because even high-confidence fields are reviewed by humans to ensure precision. See our Accuracy article for more information.
Automation reports
The Automation chart reports the percentage of automation for a given date range.
Field Transcription Automation and Table Transcription Automation percentages at the bottom of the chart report the amount of transcription automation for the selected date range. Machine Field Transcriptions report the number of machine-transcribed fields.
To see additional performance metrics, hover your cursor over the data point of interest and a window will display the percentage of automation (Transcription, Identification, or Table Identification), the number of machine-extracted fields, and the total number of fields.
Note the following about the Automation report:
For Field Identification automation – the chart reports on an occurrence level rather than on a field level.
For Table Identification automation – the percentage is composed of the number of columns correctly predicted out of the total columns that the machine predicted.
Filtering options
The following filtering options are available in the Automation chart:
Date range: by default, these figures are generated with the last week's data, but can be recalculated for a custom date range.
Task type: Transcription, Identification, or Classification data can be reported.
All Layout Variations: Choose a specific layout you want to see the automation for.
All Flows: Choose a flow you want to see the automation for.