IP Library › Granted Patent US 11,816,167
Granted Patent B1
US 11,816,167 · App. 17/343,687 · Granted Nov 14, 2023

Knowledge base platform

Inventors: Christopher Allan Doyle (Chicago, IL); Spencer Thomas Allee (Chicago, IL); Brian T. Clark (Chicago, IL)
Assignee: Ascent Technologies Inc.
G06F16/93G06N5/022
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,816,167
App. No.
17/343,687
Granted
Nov 14, 2023
Kind
B1
Abstract

An electronic computing system receives a document and identifies steps for processing the document. Each step is associated with a respective machine learning model trained to generate an output. The system subdivides content of the document into a plurality of chunks. For each step and for each chunk, the system applies the respective machine learning model to the chunk to generate an output and a confidence value. The system compares the confidence value to a threshold value. In accordance with a determination that the confidence value exceeds the threshold value, the system stores the output as final output for the chunk and the step. In accordance with a determination that the confidence value does not exceed the threshold value, the system requests user input for the chunk and the step, and the system stores received user input as final output for the chunk and step.

Claims (88)

1. A method, comprising:

at an electronic computing device having one or more processors and memory storing one or more programs configured for execution by the one or more processors:

receiving a first document;

identifying a sequential plurality of steps to process the first document, wherein each step of the plurality of steps is associated with a respective machine learning model trained to generate a respective output;

subdividing content of the first document into a first plurality of chunks; and

for each step of the plurality of steps and for each chunk of the first plurality of chunks:

determining a respective first threshold confidence value for the respective step based at least in part on a level of liability associated with the respective step;

applying the respective machine learning model to the respective chunk to generate (i) a respective output corresponding to the respective chunk and (ii) a respective confidence value for accuracy of the respective output;

comparing the respective confidence value to a respective first threshold confidence value;

in accordance with a determination that the respective confidence value exceeds the respective first threshold confidence value, storing the respective output as final output for the respective chunk and respective step; and

in accordance with a determination that the respective confidence value does not exceed the respective first threshold confidence value:

transmitting an alert requesting user input for the respective chunk and the respective step; and

in response to receiving the requested user input, storing the user input as final output for the respective chunk and respective step.

2. The method of claim 1 , wherein, for a first step of the plurality of steps:

respective output corresponding to a first subset, less than all, of the first plurality of chunks are stored as final output;

user input corresponding to a second subset, less than all, of the first plurality of chunks are stored as final output; and

the second subset of the first plurality of chunks is distinct from the first subset of the first plurality of chunks.

3. The method of claim 2 , further comprising:

receiving a second document;

subdividing content of the second document into a second plurality of chunks; and

for each step of the plurality of steps and for each chunk of the second plurality of chunks:

applying a machine learning model associated with the first step to the respective chunk to generate a respective output and to generate a respective confidence value for accuracy of the respective output;

comparing the respective confidence value to a respective second threshold confidence value;

in accordance with a determination that the respective confidence value exceeds the respective second threshold confidence value, storing the respective output as final output for the respective chunk and the first step; and

in accordance with a determination that the respective confidence value does not exceed the respective second threshold confidence value:

transmitting an alert requesting user input for the respective chunk and the first step; and

in response to receiving the requested user input, storing the user input as final output for the respective chunk and the first step.

4. The method of claim 3 , wherein the respective second threshold confidence value is different from the respective first threshold confidence value.

5. The method of claim 4 , further comprising:

using the user input as feedback to update the respective first threshold confidence value to the respective second threshold confidence value.

6. The method of claim 3 , wherein, for the first step of the plurality of steps:

respective output corresponding to a first subset, less than all, of the second plurality of chunks are stored as final output;

user input corresponding to a second subset, less than all, of the second plurality of chunks are stored as final output;

the second subset of the second plurality of chunks is distinct from the first subset of the second plurality of chunks; and

the first subset of the first plurality of chunks includes a different number of chunks than the first subset of the second plurality of chunks.

7. The method of claim 6 , wherein the first subset of the second plurality of chunks includes more chunks than the first subset of the first plurality of chunks.

8. The method of claim 3 , wherein the second plurality of chunks includes a different number of chunks than the first plurality of chunks.

9. The method of claim 2 , wherein the plurality of steps further includes a second step that follows the first step, the method further comprising:

providing final output from the first step as input to the second step.

10. The method of claim 1 , further comprising:

updating the respective machine learning model using user input for the respective step as labeled training data.

11. The method of claim 1 , wherein for a third step of the plurality of steps and for each chunk of the first plurality of chunks, the respective output is stored as final output for the third step.

12. The method of claim 1 , further comprising:

reviewing at least a sample of final output for one of the of the plurality of steps; and

providing user indication to accept the final output or to edit the final output.

13. The method of claim 1 , further comprising:

determining the respective first threshold confidence value based at least in part on a risk assurance value associated with the respective step.

14. The method of claim 13 , further comprising:

determining the risk assurance value associated with the respective step based on the level of liability associated with respective step.

15. The method of claim 1 , further comprising:

identifying one or more government regulations related to the respective step;

identifying one or more consequences of not complying with the one or more government regulations; and

determining the level of liability associated with respective step based at least in part on the one or more consequences of not complying with the one or more government regulations.

16. The method of claim 13 , further comprising:

determining an error rate associated with past executions of the respective step;

comparing the error rate with the level of liability associated with the respective step; and

determining the risk assurance value in accordance with comparing the error rate with the level of liability.

17. A computer system, comprising:

one or more databases;

an electronic device having a display and memory storing one or more programs executable by the electronic device, the one or more programs including instructions for:

receiving a first document;

identifying a sequential plurality of steps to process the first document, wherein each step of the plurality of steps is associated with a respective machine learning model trained to generate a respective output;

subdividing content of the first document into a first plurality of chunks; and

for each step of the plurality of steps and for each chunk of the first plurality of chunks:

determining a respective first threshold confidence value for the respective step based at least in part on a level of liability associated with the respective step;

applying the respective machine learning model to the respective chunk to generate (i) a respective output corresponding to the respective chunk and (ii) a respective confidence value for accuracy of the respective output;

comparing the respective confidence value to a respective first threshold confidence value;

in accordance with a determination that the respective confidence value exceeds the respective first threshold confidence value, storing the respective output as final output for the respective chunk and respective step; and

in accordance with a determination that the respective confidence value does not exceed the respective first threshold confidence value:

transmitting an alert requesting user input for the respective chunk and the respective step; and

in response to receiving the requested user input, storing the user input as final output for the respective chunk and respective step.

18. The computer system of claim 17 , wherein, for a first step of the plurality of steps:

respective output corresponding to a first subset, less than all, of the first plurality of chunks are stored as final output;

user input corresponding to a second subset, less than all, of the first plurality of chunks are stored as final output; and

the second subset of the first plurality of chunks is distinct from the first subset of the first plurality of chunks.

19. The computer system of claim 17 , wherein the one or more programs further comprise instructions for updating the respective machine learning model using user input for the respective step as labeled training data.

20. A non-transitory computer-readable storage medium having memory storing one or more programs executable by an electronic device, the one or more programs including instructions for:

receiving a first document;

identifying a sequential plurality of steps to process the first document, wherein each step of the plurality of steps is associated with a respective machine learning model trained to generate a respective output;

subdividing content of the first document into a first plurality of chunks; and

for each step of the plurality of steps and for each chunk of the first plurality of chunks:

determining a respective first threshold confidence value for the respective step based at least in part on a level of liability associated with the respective step;

applying the respective machine learning model to the respective chunk to generate (i) a respective output corresponding to the respective chunk and (ii) a respective confidence value for accuracy of the respective output;

comparing the respective confidence value to a respective first threshold confidence value;

in accordance with a determination that the respective confidence value exceeds the respective first threshold confidence value, storing the respective output as final output for the respective chunk and respective step; and

in accordance with a determination that the respective confidence value does not exceed the respective first threshold confidence value:

transmitting an alert requesting user input for the respective chunk and the respective step; and

in response to receiving the requested user input, storing the user input as final output for the respective chunk and respective step.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 27, 2021
From: DOYLE, CHRISTOPHER ALLAN; ALLEE, SPENCER THOMAS; CLARK, BRIAN T.
To: ASCENT TECHNOLOGIES INC.
Reel/Frame 057615/0992 →
Continuity (1)
Provisional Application 63036904 · Jun 9, 2020
Cited By (2)
US 12,223,354 US 12,406,215