Database for unstructured data
A technique for running queries is provided that includes a method and system for managing unstructured data and for capturing uncertain relationships between pieces of data. A structural schema is generated from unstructured data that is configured to evolve in response to user input and incoming data that is changing over the course of an application. The schema is also configured to capture relationships between data that are uncertain or difficult for a person to capture in a consistent or comprehensive manner. The technique is especially advantageous for running and returning meaningful responses to queries that require an ability to connect pieces of data received from unstructured data or disparate sources, including user input, or where the query is directed to information that is uncertain or was not anticipated as useful or relevant at the time the data containing the information was originally received.
1. A system for running queries over unstructured data, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
generate a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes;
infer a structure from the unstructured data by adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data and wherein the edge represents a relationship between two nodes in the schema, wherein the inferring of the structure comprises to:
generate the data node based on one or more of the following:
A) generate a property node based at least in part on a property associated with the data node; and
add the property node to the schema;
B) generate a property node based at least in part on a property associated with the data node;
generate a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
add the property node and the property-data edge to the schema; and/or
C) determine a property associated with the data node;
generate a property node based at least in part on the property;
generate a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
add the property, the property node, and the property-data edge to the schema;
generate an inferred node or an inferred edge that is added to the graph based at least in part on user input, wherein the inferred node or the inferred edge is based at least in part on an output of a machine learning model;
apply a model to the data node to determine a data type associated with the added data node; and
infer an attribute of the data node based on a first data type or a second data type, wherein the inferred attribute is stored as another node to the schema; and
receive a query and return a query result, wherein the query result is provided by searching the schema, and wherein the schema is configured to receive and be modified by user input.
2. The system of claim 1 , wherein the generating of the schema from the unstructured data includes to store the data extracted from the unstructured data as the data node in the graph.
3. The system of claim 1 , wherein the generating of the schema from the unstructured data includes to store the data extracted from the unstructured data as the data node in the graph.
4. The system of claim 1 , wherein at least one of the nodes is connected to another node in the plurality of nodes by an edge, wherein the edge represents a relationship between the nodes connected by the edge.
5. The system of claim 1 , wherein:
the edge includes a confidence level.
6. A method for running queries over unstructured data, comprising:
generating a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes connected by edges;
adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data, wherein the edge represents a relationship between two nodes in the schema, and wherein the adding of the data node or the edge to the schema comprises:
generating the data node based on one or more of the following:
A) generating a property node based at least in part on a property associated with the data node; and
adding the property node to the schema;
B) generating a property node based at least in part on a property associated with the data node;
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
adding the property node and the property-data edge to the schema; and/or
C) determining a property associated with the data node;
generating a property node based at least in part on the property;
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
adding the property, the property node, and the property-data edge to the schema;
generating an inferred node or an inferred edge that is added to the graph based at least in part on user input, wherein the inferred node or inferred edge is based at least in part on an output of a machine learning model;
applying a model to the data node to determine a data type associated with the added data node; and
inferring an attribute of the data node based on a first data type or a second data type, wherein the inferred attribute is stored as another node to the schema;
searching the schema in response to receiving a query; and
returning a query result based at least in part on searching the schema.
7. The method of claim 6 , further comprising receiving user input and modifying the schema based at least in part on the user input.
8. The method of claim 6 , wherein generating the schema from the unstructured data includes storing the data extracted from the unstructured data as the data node in the graph.
9. The method of claim 6 , further comprising adjusting the machine learning model based at least in part on the user input.
10. A computer program product for running queries over unstructured data, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for:
generating a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes connected by edges;
adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data, wherein the edge represents a relationship between two nodes in the schema, and wherein the adding of the data node or the edge to the schema comprises:
generating the data node based on one or more of the following:
A) generating a property node based at least in part on a property associated with the data node; and
adding the property node to the schema;
B) generating a property node based at least in part on a property associated with the data node;
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
adding the property node and the property-data edge to the schema; and/or
C) determining a property associated with the data node;
generating a property node based at least in part on the property;
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
adding the property, the property node, and the property-data edge to the schema;
generating an inferred node or an inferred edge that is added to the graph based at least in part on user input, wherein the inferred node or inferred edge is based at least in part on an output of a machine learning model;
applying a model to the data node to determine a data type associated with the added data node; and
inferring an attribute of the data node based on a first data type or a second data type, wherein the inferred attribute is stored as another node to the schema;
searching the schema in response to receiving a query; and
returning a query result based at least in part on searching the schema.