Intelligent compiler for parallel graph processing
An analyzer (such as a compiler) searches for a program portion that matches a pattern that may suffer from workload imbalance due to nodes with high degrees (i.e., relatively many edges). Such a pattern involves iteration over at least a subset (or all) of the nodes in a graph. If a program portion that matches the pattern is found, then the analyzer determines whether the body of the iteration contains an iteration over edges or neighbors of each node in the subset. If so, then the analyzer transforms the graph analytic program by adding code and, optionally, modifying existing code so that high-degree nodes are processed differently than low-degree nodes. High-degree nodes are processed sequentially while low-degree nodes are processed in parallel. Conversely, edges of high-degree nodes are processed in parallel while edges of low-degree nodes are processed sequentially.
1. A method comprising:
receiving, by a compiler, a graph analytic program that targets graph data that comprises a plurality of nodes and a plurality of edges, wherein the plurality of nodes are not executable code;
in response to receiving the graph analytic program:
identifying, by the compiler and within the graph analytic program, a code portion that involves processing edges of multiple nodes of the plurality of nodes; and
inserting, by the compiler and into the graph analytic program, additional code that, when executed with at least the code portion of the graph analytic program, causes a first subset of the multiple nodes to be processed sequentially and a second subset of the multiple nodes to be processed in parallel.
2. The method of claim 1 , wherein semantics of the code portion is a parallel iteration of the multiple nodes.
3. The method of claim 1 , wherein the additional code, when executed, causes, for each node in the first subset, multiple edges of said each node to be processed in parallel.
4. The method of claim 1 , further comprising:
inserting, into the graph analytic program, degree check code that, when executed:
determines, for each node of the multiple nodes, whether said each node has a threshold number of edges, and
for each node that has the threshold number of edges, adding said each node to a set that becomes the first subset.
5. The method of claim 4 , wherein:
the code portion includes a parallel node iteration portion and an edge iteration portion; and
the inserting the degree check code comprises inserting, into the code portion, the degree check code (1) after the parallel node iteration portion and (2) before the edge iteration portion.
6. The method of claim 4 , further comprising:
performing an analysis of the graph data to determine a number of edges of each node of the plurality of nodes; and
identifying the threshold number of edges based on the analysis.
7. The method of claim 6 , wherein:
the performing the analysis comprises analyzing the graph data to determine a subset, of the plurality of nodes, that represents a pre-defined percentage of a total number of nodes;
each node in the subset has more edges than each node, of the plurality of nodes, that is outside the subset;
the identifying the threshold number of edges comprises identifying the threshold number of edges based on a node, in the subset, that has a lowest number of edges relative to other nodes in the subset.
8. The method of claim 1 , further comprising:
after the inserting the additional code into the graph analytic program, generating, by the compiler, an executable code portion based on the graph analytic program that includes the additional code.
9. The method of claim 1 , wherein:
the code portion is a first code portion;
the method further comprising determining whether the graph analytic program includes a second code portion that is within the first code portion and that is implementable in parallel with respect to edges of each node of the multiple nodes;
the inserting the additional code into the graph analytic program comprises inserting the additional code into the graph analytic program only in response to determining that the graph analytic program includes the second code portion.
10. The method of claim 1 , wherein:
prior to the inserting the additional code into the graph analytic program, the code portion involves, for each node of the multiple nodes, sorting a set of edges of said each node.
11. The method of claim 10 , wherein:
after the inserting the additional code into the graph analytic program:
the additional code, when executed, involves a parallel sort of the edges of each node in the first subset;
the code portion, when executed, involves a sequential sort of the edges of each node in the second subset.
12. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:
receiving, by a compiler, a graph analytic program that targets graph data that comprises a plurality of nodes and a plurality of edges, wherein the plurality of nodes are not executable code;
in response to receiving the graph analytic program:
identifying, by the compiler and within the graph analytic program, a code portion that involves processing edges of multiple nodes of the plurality of nodes; and
inserting, by the compiler and into the graph analytic program, additional code that, when executed with at least the code portion of the graph analytic program, causes a first subset of the multiple nodes to be processed sequentially and a second subset of the multiple nodes to be processed in parallel.
13. The one or more non-transitory storage media of claim 12 , wherein semantics of the code portion is a parallel iteration of the multiple nodes.
14. The one or more non-transitory storage media of claim 12 , wherein the additional code, when executed, causes, for each node in the first subset, multiple edges of said each node to be processed in parallel.
15. The one or more non-transitory storage media of claim 12 , the instructions, when executed by the one or more processors, further causes:
inserting, into the graph analytic program, degree check code that, when executed:
determines, for each node of the multiple nodes, whether said each node has a threshold number of edges, and
for each node that has the threshold number of edges, adding said each node to a set that becomes the first subset.
16. The one or more non-transitory storage media of claim 15 , wherein:
the code portion includes a parallel node iteration portion and an edge iteration portion; and
the inserting the degree check code comprises inserting, into the code portion, the degree check code (1) after the parallel node iteration portion and (2) before the edge iteration portion.
17. The one or more non-transitory storage media of claim 15 , the instructions, when executed by the one or more processors, further cause:
performing an analysis of the graph data to determine a number of edges of each node of the plurality of nodes; and
identifying the threshold number of edges based on the analysis.
18. The one or more non-transitory storage media of claim 17 , wherein:
the performing the analysis comprises analyzing the graph data to determine a subset, of the plurality of nodes, that represents a pre-defined percentage of a total number of nodes;
each node in the subset has more edges than each node, of the plurality of nodes, that is outside the subset;
the identifying the threshold number of edges comprises identifying the threshold number of edges based on a node, in the subset, that has a lowest number of edges relative to other nodes in the subset.
19. The one or more non-transitory storage media of claim 12 , wherein after the inserting the additional code into the graph analytic program, generating, by the compiler, an executable code portion based on the graph analytic program that includes the additional code.
20. The one or more non-transitory storage media of claim 12 , wherein:
the code portion is a first code portion;
the method further comprising determining whether the graph analytic program includes a second code portion that is within the first code portion and that is implementable in parallel with respect to edges of each node of the multiple nodes;
the inserting the additional code into the graph analytic program comprises inserting the additional code into the graph analytic program only in response to determining that the graph analytic program includes the second code portion.
21. The one or more non-transitory storage media of claim 12 , wherein:
prior to the inserting the additional code into the graph analytic program, the code portion involves, for each node of the multiple nodes, sorting a set of edges of said each node.
22. The one or more non-transitory storage media of claim 21 , wherein: after the inserting the additional code into the graph analytic program: the additional code, when executed, involves a parallel sort of the edges of each node in the first subset; the code portion, when executed, involves a sequential sort of the edges of each node in the second subset.