Shortest and cheapest paths in distributed asynchronous graph traversals
The present disclosure relates to various approaches for fast and scalable TOP K SHORTEST and CHEAPEST graph queries supporting horizontal aggregations on the group variables of a path in a distributed graph query engine. A distributed graph query processing engine may execute a graph query in a plurality of computing devices. A plurality of subjobs may be generated based at least in part on the graph query. Execution of an asynchronous pattern matching subjob of the plurality of subjobs may be initiated, and, in response to the asynchronous pattern matching subjob identifying one or more source vertices of a plurality of vertices, the execution of the asynchronous pattern matching subjob may be paused. An output context set comprising the one or more source vertices may be generated. Execution of a synchronous path matching subjob of the plurality of subjobs may be initiated, and a reachability map may be generated based at least in part on one or more matched paths between the one or more source vertices and one or more destination vertices of the plurality of vertices. The execution of the asynchronous pattern matching subjob may be resumed based at least in part on the output context set and the reachability map.
1 . A method comprising:
a distributed graph query processing engine executing a graph query in a plurality of computing devices, wherein executing the graph query comprises:
generating a plurality of subjobs based at least in part on the graph query, the plurality of subjobs comprising an asynchronous pattern matching subjob and a synchronous path matching subjob;
initiating execution of the asynchronous pattern matching subjob of the plurality of subjobs;
in response to the asynchronous pattern matching subjob identifying one or more source vertices of a plurality of vertices, pausing the execution of the asynchronous pattern matching subjob;
generating an output context set comprising the one or more source vertices;
initiating execution of the synchronous path matching subjob of the plurality of subjobs;
generating a reachability map based at least in part on one or more matched paths between the one or more source vertices and one or more destination vertices of the plurality of vertices; and
resuming the execution of the asynchronous pattern matching subjob based at least in part on the output context set and the reachability map;
wherein the method is executed on the plurality of computing devices.
2 . The method of claim 1 , wherein individual subjobs of the plurality of subjobs comprise at least one stage, and the at least one stage corresponds to a processing of a vertex.
3 . The method of claim 1 , wherein the graph query is a SHORTEST query, and the method further comprises:
determining that a current vertex of a plurality of vertices satisfies one or more filters from the graph query;
determining that the current vertex is reachable from the one or more source vertices with a path having a same length as a current iteration number; and
identifying a new path from the one or more source vertices, through the current vertex, and to one or more neighboring vertices of the current vertex.
4 . The method of claim 1 , wherein the graph query is a CHEAPEST query, and the method further comprises:
retrieving a cheapest known path from a global priority queue, the cheapest known path having a lowest cost among one or more paths in the global priority queue; and
inserting, into the global priority queue, a new valid path from the one or more source vertices to a neighboring vertex of a last vertex in the cheapest known path.
5 . The method of claim 1 , wherein the graph query is a CHEAPEST query, and the method further comprises:
identifying one or more fresh paths from the one or more source vertices, through a current vertex, and to one or more neighboring vertices of the current vertex;
in response to determining that a fresh distance estimate associated with the one or more fresh paths improves upon one of a plurality of distance estimates from the one or more source vertices to the one or more neighboring vertices, replacing the one of the plurality of distance estimates with the fresh distance estimate; and
designating the one or more fresh paths as old.
6 . The method of claim 1 , wherein resuming the execution of the asynchronous pattern matching subjob based at least in part on the output context set and the reachability map further comprises:
identifying, in the reachability map, the one or more destination vertices corresponding to the one or more source vertices; and
performing a jump to the one or more destination vertices.
7 . The method of claim 1 , further comprising reversing the reachability map.
8 . The method of claim 1 , further comprising computing one or more horizontal aggregations for the one or more matched paths.
9 . One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:
a distributed graph query processing engine executing a graph query in a plurality of computing devices, wherein executing the graph query comprises:
generating a plurality of subjobs based at least in part on the graph query, the plurality of subjobs comprising an asynchronous patter matching subjob and a synchronous path matching subjob;
initiating execution of the asynchronous pattern matching subjob of the plurality of subjobs;
in response to the asynchronous pattern matching subjob identifying one or more source vertices of a plurality of vertices, pausing the execution of the asynchronous pattern matching subjob;
generating an output context set comprising the one or more source vertices;
initiating execution of the synchronous path matching subjob of the plurality of subjobs;
generating a reachability map based at least in part on one or more matched paths between the one or more source vertices and one or more destination vertices of the plurality of vertices; and
resuming the execution of the asynchronous pattern matching subjob based at least in part on the output context set and the reachability map.
10 . The one or more non-transitory storage media of claim 9 , wherein individual subjobs of the plurality of subjobs comprise at least one stage, and the at least one stage corresponds to a processing of a vertex.
11 . The one or more non-transitory storage media of claim 9 , wherein the graph query is a SHORTEST query, and the one or more non-transitory storage media, when executed by the one or more computing devices, further cause:
determining that a current vertex of a plurality of vertices satisfies one or more filters from the graph query;
determining that the current vertex is reachable from the one or more source vertices with a path having a same length as a current iteration number; and
identifying a new path from the one or more source vertices, through the current vertex, and to one or more neighboring vertices of the current vertex.
12 . The one or more non-transitory storage media of claim 9 , wherein the graph query is a CHEAPEST query, and the one or more non-transitory storage media, when executed by the one or more computing devices, further cause:
retrieving a cheapest known path from a global priority queue, the cheapest known path having a lowest cost among one or more paths in the global priority queue; and
inserting, into the global priority queue, a new valid path from the one or more source vertices to a neighboring vertex of a last vertex in the cheapest known path.
13 . The one or more non-transitory storage media of claim 9 , wherein the graph query is a CHEAPEST query, and the one or more non-transitory storage media, when executed by the one or more computing devices, further cause:
identifying one or more fresh paths from the one or more source vertices, through a current vertex, and to one or more neighboring vertices of the current vertex;
in response to determining that a fresh distance estimate associated with the one or more fresh paths improves upon one of a plurality of distance estimates from the one or more source vertices to the one or more neighboring vertices, replacing the one of the plurality of distance estimates with the new fresh distance estimate; and
designating the one or more fresh paths as old.
14 . The one or more non-transitory storage media of claim 9 , wherein and the one or more non-transitory storage media causing resuming the execution of the asynchronous pattern matching subjob based at least in part on the output context set and the reachability map further causes:
identifying, in the reachability map, the one or more destination vertices corresponding to the one or more source vertices; and
performing a jump to the one or more destination vertices.
15 . The one or more non-transitory storage media of claim 9 , wherein the one or more non-transitory storage media, when executed by the one or more computing devices, further cause reversing the reachability map.
16 . The one or more non-transitory storage media of claim 9 , wherein the one or more non-transitory storage media, when executed by the one or more computing devices, further cause computing one or more horizontal aggregations for the one or more matched paths.
17 . A system comprising:
one or more computing devices comprising a processor and memory;
machine-readable instructions stored in the memory that, when executed by the processor, cause the one or more computing devices to:
generate a plurality of subjobs based at least in part on a graph query, the plurality of subjobs comprising an asynchronous pattern matching subjob and a synchronous path matching subjob;
initiate execution of the asynchronous pattern matching subjob of the plurality of subjobs;
in response to the asynchronous pattern matching subjob identifying one or more source vertices of a plurality of vertices, pause the execution of the asynchronous pattern matching subjob;
generate an output context set comprising the one or more source vertices;
initiate execution of the synchronous path matching subjob of the plurality of subjobs;
generate a reachability map based at least in part on one or more matched paths between the one or more source vertices and one or more destination vertices of the plurality of vertices; and
resume the execution of the asynchronous pattern matching subjob based at least in part on the output context set and the reachability map.
18 . The system of claim 17 , wherein the graph query is a SHORTEST query, and the machine-readable instructions, when executed by the processor, further cause the one or more computing devices to:
determining that a current vertex of a plurality of vertices satisfies one or more filters from the graph query;
determining that the current vertex is reachable from the one or more source vertices with a path having a same length as a current iteration number; and
identifying a new path from the one or more source vertices, through the current vertex, and to one or more neighboring vertices of the current vertex.
19 . The system of claim 17 , wherein the graph query is a CHEAPEST query, and the machine-readable instructions, when executed by the processor, further cause the one or more computing devices to:
retrieving a cheapest known path from a global priority queue, the cheapest known path having a lowest cost among one or more paths in the global priority queue; and
inserting, into the global priority queue, a new valid path from the one or more source vertices to a neighboring vertex of a last vertex in the cheapest known path.
20 . The system of claim 17 , wherein the graph query is a CHEAPEST query, and the machine-readable instructions, when executed by the processor, further cause the one or more computing devices to:
identifying one or more fresh paths from the one or more source vertices, through a current vertex, and to one or more neighboring vertices of the current vertex;
in response to determining that a fresh distance estimate associated with the one or more fresh paths improves upon one of a plurality of distance estimates from the one or more source vertices to the one or more neighboring vertices, replacing the one of the plurality of distance estimates with the new fresh distance estimate; and
designating the one or more fresh paths as old.