Ontology-based machine learning training/validation data generation service
An ontology-based annotation service is configured to receive a customer-specified ontology that describes relationships between concepts and/or categories within a given domain, such as may be used to structure a machine learning model. The ontology-based annotation service uses the customer-specified ontology to generate annotation tools and an annotation workflow for collecting annotation data in a way such that the collected annotation data conforms to the customer-specified ontology. In some embodiments, the ontology-based annotation service uses association elements to customize system supported elements to match various different customer-specified ontologies. Additionally, the ontology-based annotation service provides a knowledge tree formatted in accordance with the customer-specified ontology and populated with annotation data structured according to the customer-specified ontology.
1 . A system, comprising:
one or more computing devices configured to:
receive, at an ontology-based annotation service, information defining an ontology for a particular type of machine learning problem;
automatically select or generate, by the ontology-based annotation service, based on the ontology, one or more annotation tools for generating annotation data;
automatically generate, by the ontology-based annotation service, based on the ontology, one or more annotation workflows for collecting the annotation data in accordance with the ontology;
coordinate, by the ontology-based annotation service, a plurality of computing resources to perform execution of the one or more annotation workflows using the one or more annotation tools to collect the annotation data in accordance with the ontology; and
automatically populate, by the ontology-based annotation service, using the plurality of computing resources, a model for the machine learning problem with the annotation data, wherein the model is structured according to the received ontology.
2 . The system of claim 1 , wherein the one or more computing devices are further configured to:
map system supported structural elements to elements of the received ontology, and
wherein to automatically populate the model for the machine learning problem with the annotation data, the plurality of computing resources implemented on the one or more computing devices are configured to:
collect, using the one or more annotation tools and the one or more annotation workflows, the annotation data in a format corresponding to a format of the received ontology; and
generate a knowledge tree with the collected annotation data formatted in accordance with the received ontology.
3 . The system of claim 1 , wherein the information defining the ontology is communicated using a shape constraint language.
4 . The system of claim 3 , wherein the annotation data is formatted using a linked data object notation.
5 . One or more non-transitory, computer-readable, media storing program instructions that, when executed using one or more processors, cause the one or more processors to:
receive, at an ontology-based annotation service, information defining an ontology;
automatically select or generate, by the ontology-based annotation service, based on the ontology, one or more annotation tools for generating annotation data;
automatically generate, by the ontology-based annotation service, based on the ontology, one or more annotation workflows for collecting the annotation data in accordance with the ontology;
coordinate, by the ontology-based annotation service, a plurality of computing resources to perform execution of the one or more annotation workflows using the one or more annotation tools to collect the annotation data in accordance with the ontology; and
automatically populate, by the ontology-based annotation service, a model with the annotation data, wherein the model is structured according to the received ontology.
6 . The one or more non-transitory, computer-readable, media of claim 5 , wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to:
generate a structure for a knowledge tree based on the received ontology, and
wherein to automatically populate the model with the annotation data, the program instructions, when executed using the one or more processors, cause the one or more processors to:
collect the annotation data in a format corresponding to a format of the knowledge tree; and
generate the knowledge tree with the collected annotation data.
7 . The one or more non-transitory, computer-readable, media of claim 6 , wherein structural elements of the knowledge tree are organized as a directed acyclic graph.
8 . The one or more non-transitory, computer-readable, media of claim 5 , wherein the information defining the ontology is formatted according to a shape constraint language.
9 . The one or more non-transitory, computer-readable, media of claim 5 , wherein the annotation data is formatted using a linked data object notation.
10 . The one or more non-transitory, computer-readable, media of claim 5 , wherein the one or more generated annotation tools comprise one or more of:
a form to be presented, via a user interface, to an annotator for use in generating the annotation data; or
a graphical user interface with which an annotator interacts to generate the annotation data.
11 . The one or more non-transitory, computer-readable, media of claim 10 , wherein the form or the graphical user interface comprises one or more of:
an element for annotating text in an image being evaluated by the annotator;
an element for identifying an object in an image being evaluated by the annotator; or
an element for identify a table in an image being evaluated by the annotator.
12 . The one or more non-transitory, computer-readable, media of claim 11 , wherein the respective elements comprise one or more sub-elements for annotating additional information about the text, the object, or the table, and
wherein the elements and the sub-elements have relationship structures in accordance with the received ontology.
13 . The one or more non-transitory, computer-readable, media of claim 5 , wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to:
cause the annotation data to be stored in a key-value database for the model.
14 . The one or more non-transitory, computer-readable, media of claim 13 , wherein the program instructions, when executed using the one or more processors, further cause the one or more processors to:
generate, based on the ontology, a graphical representation of the model in a graph-based database; and
export the annotation data from the key-value database to the graph-based database.
15 . A method, comprising:
receiving, at an ontology-based annotation service, information defining an ontology;
automatically selecting, by the ontology-based annotation service, based on the ontology, one or more annotation tools for generating annotation data;
automatically generating, by the ontology-based annotation service, based on the ontology, one or more annotation workflows for collecting the annotation data in accordance with the ontology;
coordinating, by the ontology-based annotation service, a plurality of computing resources to perform execution of the one or more annotation workflows using the one or more annotation tools to collect the annotation data in accordance with the ontology; and
automatically populating, by the ontology-based annotation service, a model with the annotation data, wherein the model is structured according to the received ontology.
16 . The method of claim 15 , further comprising:
generating a structure for a knowledge tree based on the received ontology, and wherein said automatically populating the model with the annotation data comprises:
collecting the annotation data, using the one or more annotation tools, in a format corresponding to a format of the knowledge tree; and
generating the knowledge tree with the collected annotation data.
17 . The method of claim 16 , wherein the information defining the ontology is formatted according to a shape constraint language.
18 . The method of claim 17 , wherein the annotation data is formatted using a linked data object notation.
19 . The method of claim 18 wherein the shape constraint language is in accordance with a SHACL standard and wherein the linked data object notation is JSON-LD.
20 . The method of claim 15 , further comprising:
storing the collected annotation data in a key-value database; and
exporting the collected annotation data to a graph-based database for analysis of the model.