SYSTEMS AND METHODS FOR IMPROVED ANNOTATION WORKFLOWS

1 . A computer implemented method to perform an annotation workflow on an input, the method comprising:

receiving information associated with an output from a model, the output generated based on application of the model to the input, wherein the information comprises a first concept and an associated confidence score;

identifying a first threshold model that applies to the first concept;

determining that the confidence score associated with the first concept satisfies a threshold of the first threshold model;

writing the output based on an action specified by the first threshold model; and

updating an annotation user interface based on the action specified by the first threshold model.

2 . The computer implemented method of claim 1 , wherein the action is an action to write the output with a status.

3 . The computer implemented method of claim 2 , wherein the status is one of a pending and a success status.

4 . The computer implemented method of claim 3 , wherein the status is pending, wherein updating an annotation user interface further comprises:

displaying the information associated with the output as a proposed annotation for the input; and

receiving one of an acceptance and a rejection of the proposed annotation from a user.

5 . The computer implemented method of claim 3 , wherein the status is success, further comprising:

automatically converting the information associated with the output into an annotation associated with the input.

6 . The computer implemented method of claim 5 , wherein updating an annotation user interface further comprises:

displaying the annotation to the user.

7 . The computer implemented method of claim 1 , wherein the model is a classification model.

8 . The computer implemented method of claim 1 , wherein the information associated with the output includes a second concept and an associated second confidence score.

9 . The computer implemented method of claim 8 , wherein identifying a first threshold model includes identifying a first threshold model that applies to the first concept, the method further comprising:

identifying a second threshold model that applies to the second concept;

determining that the second confidence score satisfies a threshold of the second threshold model;

writing the output based on an action specified by the second threshold model; and

updating an annotation user interface based on the action specified by the second threshold model.

10 . The computer implemented method of claim 8 , further comprising:

identifying a second threshold model and a third threshold model that apply to the second concept;

determining that the second confidence score does not satisfy a threshold of the second threshold model;

determining that the second confidence score does not satisfy a threshold of the third threshold model; and

not writing the second concept as an annotation and not updating the annotation user interface to include the second concept.

11 . The computer implemented method of claim 8 , further comprising:

identifying a second threshold model and a third threshold model that apply to the second concept;

determining that the second confidence score does not satisfy a threshold of the second threshold model;

determining that the second confidence score does satisfies a threshold of the third threshold model;

writing the second concept as a pending annotation; and

updating the annotation user interface to propose the second concept as an annotation for acceptance by a user.

12 . The computer implemented method of claim 11 , further comprising:

adding the pending annotation to a worker queue for acceptance or rejection by a user.

13 . The computer implemented method of claim 11 , further comprising:

adding the input and the pending annotation to a training data set for the model upon confirmation from the user.

14 . A system comprising:

a processing unit; and

a memory storage device including program code that when executed by the processing unit causes to the system to:

receive information associated with an output from a model, the output generated based on application of the model to the input, wherein the information comprises a first concept and an associated confidence score;

identify a first threshold model that applies to the first concept;

determine that the confidence score associated with the first concept satisfies a threshold of the first threshold model;

write the output based on an action specified by the first threshold model; and

update an annotation user interface based on the action specified by the first threshold model.

15 . The system of claim 14 , wherein the model is a classification model.

16 . The system of claim 15 , wherein the information associated with the output includes a second concept and an associated second confidence score.

17 . The system of claim 16 , wherein identifying a first threshold model includes identifying a first threshold model that applies to the first concept, the system further comprising program code that when executed by the processing unit causes to the system to:

identify a second threshold model that applies to the second concept;

determine that the second confidence score satisfies a threshold of the second threshold model;

write the output based on an action specified by the second threshold model; and

update an annotation user interface based on the action specified by the second threshold model.

18 . The system of claim 16 , further comprising program code that when executed by the processing unit causes to the system to:

identify a second threshold model and a third threshold model that apply to the second concept;

determine that the second confidence score does not satisfy a threshold of the second threshold model;

determine that the second confidence score does not satisfy a threshold of the third threshold model; and

not write the second concept as an annotation and not updating the annotation user interface to include the second concept.

19 . The system of claim 16 , further comprising program code that when executed by the processing unit causes to the system to:

identify a second threshold model and a third threshold model that apply to the second concept;

determine that the second confidence score does not satisfy a threshold of the second threshold model;

determine that the second confidence score does satisfies a threshold of the third threshold model;

write the second concept as a pending annotation; and

update the annotation user interface to propose the second concept as an annotation for acceptance by a user.