Fast distributed graph query engine
Techniques are described herein for asynchronous execution of queries on statically replicated graph data. In an embodiment, a graph is partitioned among a plurality of computers executing the graph querying engine. One or more high-degree vertices of the graph are each replicated in each graph partition. The partitions, including the replicated high-degree vertices, are loaded in memory of the plurality of computers. To execute a query, a query plan is generated based on the query. The query plan specifies a plurality of operators and an order for the plurality of operators. The order is such that if an operator requires data generated by another operator, then the other operator is ordered before the operator in the query plan. Replicated copies of a vertex is visited if matches made by subsequent operator(s) are limited by data unique to the replicated vertices.
1. A method comprising:
loading a graph into memory of a plurality of computers, the graph comprising a plurality of vertices and a plurality of edges, wherein loading the graph into memory of the plurality of computers comprises:
partitioning the graph into a plurality of graph portions, each graph portion comprising a respective set of one or more particular vertices of the plurality of vertices and a respective set of one or more edges of the plurality of edges;
identifying one or more high degree vertices of the plurality of vertices;
for each high degree vertex of the one or more high degree vertices, replicating the high degree vertex in each graph portion of the plurality of graph portions;
for each computer of the plurality of computer, loading a respective graph portion into the memory of the computer;
receiving a query for the graph;
generating, based on the plurality of graph portions, a query plan for the query, the query plan specifying a plurality of graph operations and an order for executing the plurality of graph operations;
each computer of the plurality of computers asynchronously executing one or more graph operations of the plurality of graph operations specified by same said query plan for the query that target the respective graph portion loaded into the memory of the computer.
2. The method of claim 1 wherein determining the order for executing the plurality of graph operations comprises determining a first operation requires data generated by a second operation, and wherein the order specifies executing the second operation prior to the first operation.
3. The method of claim 1 wherein the order for executing the plurality of graph operations is applied to the entire graph.
4. The method of claim 1 wherein the order for executing the plurality of graph operations is applied to the respective graph portion loaded into memory of each computer of the plurality of computers.
5. The method of claim 1 wherein executing the one or more graph operations comprises, for each graph operation, determining whether to request data from one or more replicated vertices.
6. The method of claim 5 wherein determining whether to request data from one or more replicated vertices comprises determining whether one or more subsequent operations are limited by data unique to each of the one or more replicated vertices.
7. The method of claim 5 wherein determining whether to request data from one or more replicated vertices is based on an operation type of the graph operation.
8. The method of claim 7 wherein the operation type of the graph operation is a neighbor match, wherein a particular vertex matched by the neighbor match is a replicated vertex, the method further comprising determining data should be requested from copies of the particular vertex at one or more other computers.
9. The method of claim 7 wherein the operation type of the graph operation is an edge match, wherein executing the graph operation comprises determining whether one or more edges of a first particular vertex connect to a second particular vertex, wherein the first particular vertex is a replicated vertex, the method further comprising determining data should be requested from copies of the first particular vertex at one or more other computers.
10. The method of claim 7 wherein the operation type of the graph operation is a common neighbor match, wherein executing the common neighbor match comprises determining whether a first particular vertex and a second particular vertex are connected to visited by the common neighbor match is a replicated vertex, the method further comprising determining data should be requested from copies of the first particular vertex and copies of the second particular vertex at one or more other computers.
11. A system comprising:
one or more processors;
non-transitory computer-readable media storing instructions which, when executed by the one or more processors, causes:
loading a graph into memory of a plurality of computers, the graph comprising a plurality of vertices and a plurality of edges, wherein loading the graph into memory of the plurality of computers comprises:
partitioning the graph into a plurality of graph portions, each graph portion comprising a respective set of one or more particular vertices of the plurality of vertices and a respective set of one or more edges of the plurality of edges;
identifying one or more high degree vertices of the plurality of vertices;
for each high degree vertex of the one or more high degree vertices, replicating the high degree vertex in each graph portion of the plurality of graph portions;
for each computer of the plurality of computer, loading a respective graph portion into the memory of the computer;
receiving a query for the graph;
generating, based on the plurality of graph portions, a query plan for the query, the query plan specifying a plurality of graph operations and an order for executing the plurality of graph operations;
each computer of the plurality of computers asynchronously executing one or more graph operations of the plurality of graph operations specified by same said query plan for the query that target the respective graph portion loaded into the memory of the computer.
12. The system of claim 11 wherein determining the order for executing the plurality of graph operations comprises determining a first operation requires data generated by a second operation, and wherein the order specifies executing the second operation prior to the first operation.
13. The system of claim 11 wherein the order for executing the plurality of graph operations is applied to the entire graph.
14. The system of claim 11 wherein the order for executing the plurality of graph operations is applied to the respective graph portion loaded into memory of each computer of the plurality of computers.
15. The system of claim 11 wherein executing the one or more graph operations comprises, for each graph operation, determining whether to request data from one or more replicated vertices.
16. The system of claim 15 wherein determining whether to request data from one or more replicated vertices comprises determining whether one or more subsequent operations are limited by data unique to each of the one or more replicated vertices.
17. The system of claim 15 wherein determining whether to request data from one or more replicated vertices is based on an operation type of the graph operation.
18. The system of claim 17 wherein the operation type of the graph operation is a neighbor match, wherein a particular vertex matched by the neighbor match is a replicated vertex, the instructions further cause determining data should be requested from copies of the particular vertex at one or more other computers.
19. The system of claim 17 wherein the operation type of the graph operation is an edge match, wherein executing the graph operation comprises determining whether one or more edges of a first particular vertex connect to a second particular vertex, wherein the first particular vertex is a replicated vertex, the instructions further cause determining data should be requested from copies of the first particular vertex at one or more other computers.
20. The system of claim 17 wherein the operation type of the graph operation is a common neighbor match, wherein executing the common neighbor match comprises determining whether a first particular vertex and a second particular vertex are connected to visited by the common neighbor match is a replicated vertex, the instructions further cause determining data should be requested from copies of the first particular vertex and copies of the second particular vertex at one or more other computers.