DATABASE FOR UNSTRUCTURED DATA
A technique for running queries is provided that includes a method and system for managing unstructured data and for capturing uncertain relationships between pieces of data. A structural schema is generated from unstructured data that is configured to evolve in response to user input and incoming data that is changing over the course of an application. The schema is also configured to capture relationships between data that are uncertain or difficult for a person to capture in a consistent or comprehensive manner. The technique is especially advantageous for running and returning meaningful responses to queries that require an ability to connect pieces of data received from unstructured data or disparate sources, including user input, or where the query is directed to information that is uncertain or was not anticipated as useful or relevant at the time the data containing the information was originally received.
1 . (canceled)
2 . A system for running queries over unstructured data, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
generate a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes;
infer a structure from the unstructured data by adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data, wherein the edge represents a relationship between two nodes in the schema, and wherein the inferring of the structure comprises to:
perform one or more of the following:
A) generate a property node based at least in part on a property associated with the data node;
B) generate a property node based at least in part on a property associated with the data node; and
generate a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and/or
C) determine a property associated with the data node;
generate a property node based at least in part on an attribute; and
generate a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and
receive a query and return a query result, wherein the query result is provided by searching the schema, wherein the schema is configured to receive and be modified by user input.
3 . The system of claim 2 , wherein generating the schema from the unstructured data includes storing the data extracted from the unstructured data as the data node in the graph.
4 . The system of claim 2 , wherein the property is a data type.
5 . The system of claim 2 , wherein the processor is further configured to apply a model to the data node to determine the property associated with the data node.
6 . The system of claim 2 , wherein:
generating the schema from the unstructured data includes storing the data extracted from the unstructured data as the data node in the graph; and
the processor is further configured to generate an inferred node that is added to the graph based at least in part on the data node or user input.
7 . The system of claim 2 , wherein at least one of the nodes is connected to another node in the plurality of nodes by an edge, wherein the edge represents a relationship between the nodes connected by the edge.
8 . The system of claim 2 , wherein the processor is further configured to generate an inferred node or an inferred edge that is added to the graph based at least in part on user input.
9 . The system of claim 2 , wherein:
the processor is further configured to generate an inferred node or an inferred edge that is added to the graph based at least in part on user input; and
the inferred node or inferred edge is based at least in part on an output of a machine learning model.
10 . The system of claim 9 , wherein the processor is further configured to adjust the machine learning model based at least in part on the user input.
11 . The system of claim 2 , wherein:
the processor is further configured to generate an inferred node or an inferred edge that is added to the graph based at least in part on user input;
the inferred node or inferred edge is based at least in part on an output of a machine learning model; and
the edge includes a confidence level.
12 . A method for running queries over unstructured data, comprising:
generating a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes connected by edges;
inferring a structure from the unstructured data by adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data, wherein the edge represents a relationship between two nodes in the schema, and wherein the inferring of the structure comprises:
performing one or more of the following:
A) generating a property node based at least in part on a property associated with the data node;
B) generating a property node based at least in part on a property associated with the data node; and
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and/or
C) determining a property associated with the data node;
generating a property node based at least in part on an attribute; and
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node;
searching the schema in response to receiving a query; and
returning a query result based at least in part on searching the schema.
13 . The method of claim 12 , further comprising receiving user input and modifying the schema based at least in part on the user input.
14 . The method of claim 12 comprising generating an inferred node or an inferred edge that is added to the graph based at least in part on user input.
15 . The method of claim 12 comprising generating an inferred node or an inferred edge that is added to the graph based at least in part on user input, wherein the inferred node or inferred edge is based at least in part on an output of a machine learning model.
16 . The method of claim 15 , further comprising adjusting the machine learning model based at least in part on the user input.
17 . The method of claim 12 , further comprising:
receiving user input;
extracting additional data in response to the user input; and
modifying the schema based at least in part on the user input, including by processing the additional data to generate one or more additional nodes, wherein the one or more additional nodes can be linked to another node in the graph via an edge.
18 . A computer program product for running queries over unstructured data, the computer program product being embodied in a tangible computer readable storage medium and comprising computer instructions for:
generating a schema from the unstructured data, wherein the schema comprises a graph having a plurality of nodes connected by edges;
inferring a structure from the unstructured data by adding a data node or an edge to the schema, wherein the data node or the edge is based at least in part on data extracted from the unstructured data wherein the edge represents a relationship between two nodes in the schema, and wherein the inferring of the structure comprises:
performing one or more of the following:
A) generating a property node based at least in part on a property associated with the data node;
B) generating a property node based at least in part on a property associated with the data node; and
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node; and/or
C) determining a property associated with the data node;
generating a property node based at least in part on an attribute; and
generating a property-data edge between the property node and the data node, wherein the property-data edge represents a relationship between the property node and the data node;
searching the schema in response to receiving a query; and
returning a query result based at least in part on searching the schema.