System and method for hydrating graph databases from external data
One example method includes receiving, by a graph proxy, an event from an event generator, and the event includes information about an IO and information about data affected by the IO, comparing, by the graph proxy, the data to a schema, when the data is determined by the graph proxy to map to the schema, identifying, by the graph proxy, a rule that is associated with the event, and the rule specifies performance of an action when a condition is met, and when the condition is met, performing, by the graph proxy, the action, and the action is performed with respect to a graph.
1. A method, comprising:
listening in real time, by a graph proxy hosted at a server accessible by event generators, one or more of which comprises an application, file server, or object store, for input/output (IO) operations incoming from the event generators;
receiving in real time, by the graph proxy from each of the event generators, a respective continuous stream of events, wherein one of the continuous streams includes metadata of an event that comprises information about one of the IO operations and information about data to which the IO operation is directed;
storing the respective continuous stream of events in a buffer;
determining by the graph proxy that the metadata includes a schema, and identifying, by the graph proxy, a rule that is associated with the event, wherein the rule includes an edge rule and a vertex rule, each of which specifies performance of an action;
determining that a condition of the vertex or edge rule is met in the metadata;
creating and adding to a graph a vertex with a property of the metadata;
determining that the condition of the edge rule is met by the metadata, and creating and adding an edge to the graph, and the edge connects two vertices and includes a property of the edge rule, wherein the property of the edge rule identifies a relationship between the two vertices; and
when the edge is created, overlaying, in the graph, properties of metadata corresponding to the two vertices over the two vertices and the property of the edge rule over the edge.
2. The method as recited in claim 1 , wherein the action comprises one of creating, updating, or deleting, an element of the graph.
3. The method as recited in claim 1 , wherein the action is performed with respect to an element of the graph, and the element of the graph comprises one of a vertex, an edge, a property of a vertex, or a property of an edge.
4. The method as recited in claim 3 , wherein the element is a property, and the action performed by the graph proxy comprises modifying the property.
5. The method as recited in claim 1 , wherein the schema is a parent schema that references a child schema, and the data matches the parent schema and the child schema.
6. The method as recited in claim 1 , wherein the action comprises creating an edge of the graph, and the method further comprises specifying a property for inclusion in the edge.
7. The method as recited in claim 1 , wherein the schema is identified by implication.
8. The method as recited in claim 1 , wherein the schema is identified by an association process in which a schema is derived from the data, and the schema to which the data is compared is the derived schema.
9. The method as recited in claim 1 , further comprising creating, a priori based on another rule, an edge in the graph.
10. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
listening in real time, by a graph proxy hosted at a server accessible by event generators, one or more of which comprises an application, file server, or object store, for input/output (IO) operations incoming from the event generators;
receiving in real time, by the graph proxy from each of the event generators, a respective continuous stream of events, wherein one of the continuous streams includes metadata of an event that comprises information about one of the IO operations and information about data to which the IO operation is directed;
storing the respective continuous stream of events in a buffer;
when the metadata is determined by the graph proxy to include a schema, identifying, by the graph proxy, a rule that is associated with the event, wherein the rule includes an edge rule and a vertex rule, each of which specifies performance of an action;
determining whether or not a condition of the vertex or edge rule is met in the metadata;
when it is determined that the condition of the vertex rule is met by the metadata, creating and adding to a graph a vertex with a property of the metadata;
when it is determined that the condition of the edge rule is met by the metadata, creating and adding to the graph an edge, which connects two vertices and includes a property of the edge rule, wherein the property of the edge rule identifies a relationship between the two vertices; and
when the edge is created, overlaying, in the graph, properties of metadata corresponding to the two vertices over the two vertices and the property of the edge rule over the edge.
11. The non-transitory storage medium as recited in claim 10 , wherein the action comprises one of creating, updating, or deleting, an element of the graph.
12. The non-transitory storage medium as recited in claim 10 , wherein the action is performed with respect to an element of the graph, and the element of the graph comprises one of a vertex, an edge, a property of a vertex, or a property of an edge.
13. The non-transitory storage medium as recited in claim 12 , wherein the element is a property, and the action performed by the graph proxy comprises modifying the property.
14. The non-transitory storage medium as recited in claim 10 , wherein the schema is a parent schema that references a child schema, and the data matches the parent schema and the child schema.
15. The non-transitory storage medium as recited in claim 10 , wherein the action comprises creating an edge of the graph, and the operations further comprise specifying a property for inclusion in the edge.
16. The non-transitory storage medium as recited in claim 10 , wherein the action is performed automatically when the condition is met by the schema and the event.
17. The non-transitory storage medium as recited in claim 10 , wherein the schema is identified by implication.
18. The non-transitory storage medium as recited in claim 10 , wherein the schema is identified by an association process in which a schema is derived from the data, and the schema to which the data is compared is the derived schema.
19. The non-transitory storage medium as recited in claim 10 , further comprising creating, a priori based on another rule, an edge in the graph.