Graphical systems and methods for human-in-the-loop machine intelligence

1. A method comprising:

accessing first inputs managed through a first graphic user interface operated by a project manager, wherein accessing first inputs comprises:

accessing at least one document sourced by a third party user;

accessing at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and

accessing at least one first guideline describing the at least one first label;

constructing a first work unit interface, wherein constructing the first work unit interface comprises:

assigning a document among the at least one document to a document pane within the first work unit interface;

assigning the at least one first label to a label pane within the first work unit interface;

generating a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of the at least one first label for the document within the classification ontology;

assigning the generated human readable prompt to a prompt pane within the first work unit interface;

pairing a single first guideline from the at least one first guideline with a single label of the at least one first label;

creating a reference button for each single first guideline paired with a single label; and

placing the reference button adjacent to the paired label within the label pane of the first work unit interface;

displaying to at least one expert annotator operating a second graphic user interface the first work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single first guideline;

accessing a second input from the second graphic user interface operated by the expert annotator, wherein the second input comprises at least one second guideline describing the at least one first label;

constructing a second work unit interface, wherein constructing the second work unit interface comprises:

assigning the document to a document pane within the second work unit interface;

assigning the at least one first label to a label pane within the second work unit interface;

assigning the generated human readable prompt to a prompt pane within the second work unit interface;

pairing a single second guideline from the at least one second guideline with the single label of the at least one first label;

creating a reference button for each single second guideline paired with a single label; and

placing the reference button for each single second guideline adjacent to the paired label within the label pane of the second work unit interface;

displaying to at least one annotator operating a third graphic user interface the second work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single second guideline;

receiving at least one annotation on the second work unit interface displayed on the third graphic user interface; and

aggregating the at least one annotation on the first graphic user interface operated by the project manager.

2. The method of claim 1 , wherein aggregating the at least one annotation further comprises:

constructing an annotation agreement interface, wherein constructing an annotation agreement interface further comprises:

creating a label feedback pane, wherein the label feedback pane is configured to display at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label;

creating a learning curve pane, wherein the learning curve pane is configured to display a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology;

creating an annotation feedback pane, wherein the annotation feedback pane is configured to display at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and

creating a rules pane, wherein the rules pane is configured to display at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane.

3. The method of claim 2 , further comprising computing a plurality of initial annotation agreements from the aggregated annotations.

4. The method of claim 3 , further comprising populating the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane with computed initial annotation agreements.

5. The method of claim 4 , further comprising displaying the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface.

6. The method of claim 5 , further comprising:

receiving a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;

preparing at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and

displaying on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.

7. The method of claim 6 , further comprising:

unpairing the single first guideline paired with the single label on the first work unit interface;

pairing the single second guideline with the unpaired single label on the first work unit interface;

creating a second reference button for the paired single second guideline paired with the single label; and

displaying the second reference button within the first work unit interface's label pane adjacent to the single label.

8. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:

accessing first inputs managed through a first graphic user interface operated by a project manager, wherein accessing first inputs comprises:

accessing at least one document sourced by a third party user;

accessing at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and

accessing at least one first guideline describing the at least one first label;

constructing a first work unit interface, wherein constructing the first work unit interface comprises:

assigning a document among the at least one document to a document pane within the first work unit interface;

assigning the at least one first label to a label pane within the first work unit interface;

generating a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of the at least one first label for the document within the classification ontology;

assigning the generated human readable prompt to a prompt pane within the first work unit interface;

pairing a single first guideline from the at least one first guideline with a single label of the at least one first label;

creating a reference button for each single first guideline paired with a single label; and

placing the reference button adjacent to the paired label within the label pane of the first work unit interface;

displaying to at least one expert annotator operating a second graphic user interface the first work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single first guideline;

accessing a second input from the second graphic user interface operated by the expert annotator, wherein the second input comprises at least one second guideline describing the at least one first label;

constructing a second work unit interface, wherein constructing the second work unit interface comprises:

assigning the document to a document pane within the second work unit interface;

assigning the at least one first label to a label pane within the second work unit interface;

assigning the generated human readable prompt to a prompt pane within the second work unit interface;

pairing a single second guideline from the at least one second guideline with the single label of the at least one first label;

creating a reference button for each single second guideline paired with a single label; and

placing the reference button for each single second guideline adjacent to the paired label within the label pane of the second work unit interface;

displaying to at least one annotator operating a third graphic user interface the second work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single second guideline;

receiving at least one annotation on the second work unit interface displayed on the third graphic user interface; and

aggregating the at least one annotation on the first graphic user interface operated by the project manager.

9. The computer readable medium of claim 8 , wherein the operations to aggregate the at least one annotation further comprise:

constructing an annotation agreement interface, wherein constructing an annotation agreement interface further comprises:

creating a label feedback pane, wherein the label feedback pane is configured to display at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label;

creating a learning curve pane, wherein the learning curve pane is configured to display a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology;

creating an annotation feedback pane, wherein the annotation feedback pane is configured to display at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and

creating a rules pane, wherein the rules pane is configured to display at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane.

10. The computer readable medium of claim 9 , wherein the operations further comprise:

computing a plurality of initial annotation agreements from the aggregated annotations.

11. The computer readable medium of claim 10 , wherein the operations further comprise:

populating the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane with computed initial annotation agreements.

12. The computer readable medium of claim 11 , wherein the operations further comprise:

displaying the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface.

13. The computer readable medium of claim 12 , wherein the operations further comprise:

receiving a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;

preparing at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and

displaying on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.

14. The computer readable medium of claim 13 , wherein the operations further comprise:

unpairing the single first guideline paired with the single label on the first work unit interface;

pairing the single second guideline with the unpaired single label on the first work unit interface;

creating a second reference button for the paired single second guideline paired with the single label; and

displaying the second reference button within the first work unit interface's label pane adjacent to the single label.

15. An interface integration system comprising:

a data processor;

an input and output module from at least one of a first graphic user interface associated with a project manager, a second graphic user interface operated by an expert annotator, and a third graphic user interface operated by an annotator;

a natural language modeling engine operably coupled to the input and output module, configured to execute instructions received from the data processor to:

access first inputs managed by the first graphic user interface, wherein the access to first inputs further comprises:

access at least one document sourced by a third party user;

access at least one first label associated with the at least one document, wherein the at least one first label is selected from a classification ontology built from a collection comprising the at least one document; and

access at least one first guideline describing the at least one first label;

construct a first work unit interface, wherein to construct the first work unit interface the natural language modeling engine is further configured to:

assign a document among the at least one document to a document pane within the first work unit interface;

assign the at least one first label to a label pane within the first work unit interface;

generate a human readable prompt requesting a task of the document, wherein the task is a confirmation of the accuracy for classification of the at least one first label for the document within the classification ontology;

assign the generated human readable prompt to a prompt pane within the first work unit interface;

pair a single first guideline from the at least one first guideline with a single label of the at least one first label;

create a reference button for each single first guideline paired with a single label; and

place the reference button adjacent to the paired label within the label pane of the first work unit interface;

display to at least one expert annotator operating the second graphic user interface the first work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single first guideline;

access a second input from the second graphic user interface operated by the expert annotator, wherein the second input comprises at least one second guideline describing the at least one first label;

construct a second work unit interface, wherein to construct the second work unit interface the natural language modeling engine is further configured to:

assign the document to a document pane within the second work unit interface;

assign the at least one first label to a label pane within the second work unit interface;

assign the generated human readable prompt to a prompt pane within the second work unit interface;

pair a single second guideline from the at least one second guideline with the single label of the at least one first label;

create a reference button for each single second guideline paired with a single label; and

place the reference button for each single second guideline adjacent to the paired label within the label pane of the second work unit interface;

display to at least one annotator operating the third graphic user interface the second work unit interface comprising each of the document, the human readable prompt, the at least one first label, and the reference button for each single second guideline;

receive at least one annotation on the second work unit interface displayed on the third graphic user interface; and

aggregate the at least one annotation in an annotation agreement interface of the first graphic user interface associated with the project manager.

16. The interface integration system of claim 15 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:

construct an annotation agreement interface, wherein the instructions to construct an annotation agreement interface further comprises instructions to:

create a label feedback pane, wherein the instructions to create a label feedback pane configure the display of at least one from the group comprising a plurality of label panes and each label pane comprising a description of a label, an indicator of the number of annotated documents associated with the label, a button to delete the label from the classification ontology, and a button to edit the label;

create a learning curve pane, wherein the instructions to create a learning curve pane configure the display of a graphical representation of the relationship among a number of annotations aggregated on the first graphic user interface, an agreement among annotators of a label or task of a document, and an accuracy of the classification ontology;

create an annotation feedback pane, wherein the instructions to create an annotation feedback pane configured the display of at least one of the group comprising a collection annotation agreement score, an individual annotator agreement list, a suggested label collapse list, an agreement per label graphical representation, a collapsed agreement per label graphical representation, and a per document agreement list; and

create a rules pane, wherein the instructions to create a rules pane configure the display of at least one of the group comprising a phrase pane, a weighting adjustment pane, and an add rule pane.

17. The interface integration system of claim 16 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:

compute a plurality of initial annotation agreements from the aggregated annotations; and

populate the annotation agreement interface's label feedback pane, learning curve pane, and annotation feedback pane.

18. The interface integration system of claim 17 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions from the data processor to display the plurality of initial annotation agreements on the annotation agreement interface of the first graphic user interface.

19. The interface integration system of claim 18 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:

receive a request through the annotation agreement interface of the first graphic user interface to collapse at least one first label or task into at least one second label or task;

prepare at least one subsequent annotation agreement as computed from the initial annotation agreement of the first label or task and second label or task collapsed into one another; and

display on the annotation agreement interface of the first graphic user interface the at least one subsequent annotation agreement among the plurality of initial annotation agreements.

20. The interface integration system of claim 19 , wherein the natural language modeling engine operably coupled to the input and output module is further configured to execute instructions received from the data processor to:

unpair the single first guideline paired with the single label on the first work unit interface;

pair the single second guideline with the unpaired single label on the first work unit interface;

create a second reference button for the paired single second guideline paired with the single label; and

display the second reference button within the first work unit interface's label pane adjacent to the single label.