Human-augmented artificial intelligence configuration and optimization insights
A computer-implemented method includes receiving a document insight request that requests document insights for a corpus of documents. The document insight request includes the corpus of documents, a set of entities contained within each document of the corpus of documents, and document insight request parameters that includes a confidence value threshold. The method also includes generating the document insights for the corpus of documents based on the confidence value threshold. Here, the document insights include an accuracy target and a user review rate target. The method also includes transmitting the document insights to the user device causing a graphical user interface to display the document insights on the user device.
1. A computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations comprising:
receiving a document insight request requesting document insights for a corpus of documents from a user device associated with a user, the document insight request comprising:
the corpus of documents;
a set of entities contained within each document of the corpus of documents; and
a document insight request parameter comprising a confidence value target;
obtaining a machine learning model based on the corpus of documents, the machine learning model trained on a training corpus of documents containing the set of entities;
generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, the document insights for the corpus of documents comprising an accuracy target indicating a ratio of correctly identified entities from the corpus of documents and a user review rate target corresponding to the accuracy target; and
transmitting the document insights to the user device, the document insights, when received by the user device, causing a graphical user interface (GUI) executing on the user device to display the document insights on a display screen of the user device.
2. The computer-implemented method of claim 1 , wherein, for each respective entity of the set of entities, the document insights for the corpus of documents further comprises:
an entity-level accuracy target indicating a ratio of correctly identified entities for the respective entity; and
an entity-level user review rate target corresponding to the entity-level accuracy target.
3. The computer-implemented method of claim 1 , wherein the operations further comprise:
extracting target features from the corpus of documents, the extracted target features corresponding to the set of entities and suitable for input to the machine learning model,
wherein the machine learning model is configured to:
receive the set of extracted target features as input; and
generate the document insights for the corpus of documents as output.
4. The computer-implemented method of claim 1 , wherein the operations further comprise:
determining, using the machine learning model, confidence values for the set of entities based on the training corpus of documents; and
comparing the determined confidence values to the confidence value target to generate the user review rate target,
wherein the user review rate target indicates a ratio of documents in the corpus of documents having a confidence value that fails to satisfy the confidence value target.
5. The computer-implemented method of claim 4 , wherein the operations further comprise:
determining, using the machine learning model, accuracy estimates for the set of entities based on the training corpus of documents; and
generating the accuracy target by comparing the accuracy estimates for the set of entities and the ratio of documents in the corpus of documents having the confidence values that fail to satisfy the confidence value target.
6. The computer-implemented method of claim 1 , wherein the document insight request further comprises a document type indicator specifying a document type for the corpus of documents.
7. The computer-implemented method of claim 6 , wherein the obtaining the machine learning model further comprises selecting the machine learning model from a plurality of machine learning models based on the document type indicator.
8. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a document insight request requesting document insights for a corpus of documents from a user device associated with a user, the document insight request comprising:
the corpus of documents;
a set of entities contained within each document of the corpus of documents; and
a document insight request parameter comprising a confidence value target;
obtaining a machine learning model based on the corpus of documents, the machine learning model trained on a training corpus of documents containing the set of entities,
generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, the document insights for the corpus of documents comprising an accuracy target indicating a ratio of correctly identified entities from the corpus of documents and a user review rate target corresponding to the accuracy target; and
transmitting the document insights to the user device, the document insights, when received by the user device, causing a graphical user interface (GUI) executing on the user device to display the document insights on a display screen of the user device.
9. The system of claim 8 , wherein, for each respective entity of the set of entities, the document insights for the corpus of documents further comprises:
an entity-level accuracy target indicating a ratio of correctly identified entities for the respective entity; and
an entity-level user review rate target corresponding to the entity-level accuracy target.
10. The system of claim 8 , wherein the operations further comprise:
extracting target features from the corpus of documents, the extracted target features corresponding to the set of entities and suitable for input to the machine learning model,
wherein the machine learning model is configured to:
receive the set of extracted target features as input; and
generate the document insights for the corpus of documents as output.
11. The system of claim 8 , wherein the accuracy target indicates a ratio of correctly identified entities from the corpus of documents and the user review rate target corresponds to the accuracy target.
12. The system of claim 8 , wherein the operations further comprise:
determining, using the machine learning model, confidence values for the set of entities based on the training corpus of documents; and
comparing the determined confidence values to the confidence value target to generate the user review rate target,
wherein the user review rate target indicates a ratio of documents in the corpus of documents having a confidence value that fails to satisfy the confidence value target.
13. The system of claim 12 , wherein the operations further comprise:
determining, using the machine learning model, accuracy estimates for the set of entities based on the training corpus of documents; and
generating the accuracy target by comparing the accuracy estimates for the set of entities and the ratio of documents in the corpus of documents having the confidence values that fail to satisfy the confidence value target.
14. The system of claim 8 , wherein the document insight request further comprises a document type indicator specifying a document type for the corpus of documents.
15. The system of claim 14 , wherein the obtaining the machine learning model further comprises selecting the machine learning model from a plurality of machine learning models based on the document type indicator.
16. A computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations comprising:
receiving a document insight request requesting document insights for a corpus of documents from a user device associated with a user, the document insight request comprising:
the corpus of documents;
a set of entities contained within each document of the corpus of documents; and
a document insight request parameter comprising an accuracy target;
obtaining a machine learning model based on the corpus of documents, the machine learning model trained on a training corpus of documents containing the set of entities;
generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, the document insights for the corpus of documents comprising a confidence value target and a user review rate target, wherein generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, comprises:
generating, using the machine learning model, the user review rate target based on the training corpus of documents and the accuracy target; and
generating, using the machine learning model, the confidence value target based on the training corpus of documents,
wherein the confidence value target and the user review rate target are necessary to satisfy the accuracy target for the corpus of documents; and
transmitting the document insights to the user device, the document insights, when received by the user device, causing a graphical user interface (GUI) executing on the user device to display the document insights on a display screen of the user device.
17. The computer-implemented method of claim 16 , wherein the operations further comprise:
receiving a user input adjusting the confidence value target generated by the machine learning model; and
in response to receiving the user input, updating the confidence value target and the user review rate target to satisfy the adjusted confidence value target for the corpus of documents.
18. A computer-implemented method that, when executed by data processing hardware, causes the data processing hardware to perform operations comprising:
receiving a document insight request requesting document insights for a corpus of documents from a user device associated with a user, the document insight request comprising:
the corpus of documents;
a set of entities contained within each document of the corpus of documents; and
a document insight request parameter comprising a user review rate target;
obtaining a machine learning model based on the corpus of documents, the machine learning model trained on a training corpus of documents containing the set of entities;
generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, the document insights for the corpus of documents comprising a confidence value target and an accuracy target, wherein generating, using the machine learning model, the document insights for the corpus of documents based on the document insight request parameter, comprises:
generating, by the machine learning model, the confidence value target based on the training corpus of documents; and
generating, by the machine learning model, the accuracy target based on the training corpus of documents and the user review rate target,
wherein the confidence value target and the accuracy target are necessary to satisfy the accuracy target for the corpus of documents; and
transmitting the document insights to the user device, the document insights, when received by the user device, causing a graphical user interface (GUI) executing on the user device to display the document insights on a display screen of the user device.