Data backup in a graph processing system
Data are received at a worker system in a distributed computing system that describe a graph representing relationships among a set of items. The graph models a condition having an associated problem. The graph has graph components having associated data fields. The received data are stored in a backup table, and the relationships are analyzed to identify a solution to the problem. As part of the analysis, a new value for the data field associated with a graph component is identified and compared with an existing value of the data field, and the data field is modified. The modified data field is stored in a delta table representing a change to the backup table.
1. A computer-implemented method executed by one or more processors for analyzing graph data representing relationships among items, the graph data including a plurality of vertices connected by one or more edges representing the relationships among items in the graph, each of the vertices associated with one or more data fields, the method comprising:
identifying requests by a worker system in a distributed computing system including a plurality of workers systems to which the graph data is partitioned, each request to change a data field associated with a particular vertex in the graph data responsive to an operation performed during an analysis of the graph data, the request including a new value for the data field, and for each request:
comparing, by the worker system, the new value with an existing value of the data field associated with the particular vertex in the graph data;
modifying, by the worker system, the data field in the graph data based on the comparison, wherein modifying the graph data does not include modifying backup data for the graph data stored in a backup table; and
responsive to modification of the data field in the graph data, storing, by the worker system in a memory of the worker system, a representation of the modified data field in a respective delta table for the request, the respective delta table different than the backup table;
determining, by the worker system, after each request, whether a number of respective delta tables has reached a merge threshold;
in response to determining that the number of respective delta tables has reached the merge threshold:
merging, by the worker system, the backup table and the delta tables to create a new backup table; and
replacing, by the worker system, the backup table with the new backup table.
2. The method of claim 1 , further comprising:
responsive to an additional modification of the data field in the graph data after storing the representation of the modified data field in the delta table, storing an additional representation of the modified data field in an additional delta table.
3. The method of claim 1 , wherein comparing the new value and modifying the data field are repeated for a plurality of iterations, and the method further comprising:
storing a representation of the modified data field from each iteration in a different delta table.
4. The method of claim 3 , wherein storing the representation of the modified data field from each iteration in a different delta table is performed in response to the data field already existing in the delta table.
5. The method of claim 3 , further comprising:
merging the backup table and the delta tables created in the plurality of iterations after a threshold number of iterations have been completed.
6. The method of claim 1 , wherein the backup data for the graph data is a copy of the graph data.
7. A non-transitory, computer-readable medium storing instructions operable when executed to cause at least one processor to perform operations for analyzing graph data representing relationships among items, the graph data including a plurality of vertices connected by one or more edges representing the relationships among items in the graph, each of the vertices associated with one or more data fields, the operations comprising:
identifying requests by a worker system in a distributed computing system including a plurality of workers systems to which the graph data is partitioned, each request to change a data field associated with a particular vertex in the graph data responsive to an operation performed during an analysis of the graph data, the request including a new value for the data field, and for each request:
comparing, by the worker system, the new value with an existing value of the data field associated with the particular vertex in the graph data;
modifying, by the worker system, the data field in the graph data based on the comparison, wherein modifying the graph data does not include modifying backup data for the graph data stored in a backup table; and
responsive to modification of the data field in the graph data, storing, by the worker system in a memory of the worker system, a representation of the modified data field in a respective delta table for the request, the respective delta table different than the backup table;
determining, by the worker system, after each request, whether a number of respective delta tables has reached a merge threshold;
in response to determining that the number of respective delta tables has reached the merge threshold:
merging, by the worker system, the backup table and the delta tables to create a new backup table; and
replacing, by the worker system, the backup table with the new backup table.
8. The non-transitory computer-readable medium of claim 7 , the operations further comprising:
responsive to an additional modification of the data field in the graph data after storing the representation of the modified data field in the delta table, storing an additional representation of the modified data field in an additional delta table.
9. The non-transitory computer-readable medium of claim 7 , wherein comparing the new value and modifying the data field are repeated for a plurality of iterations, and the operations further comprising:
storing a representation of the modified data field from each iteration in a different delta table.
10. The non-transitory computer-readable medium of claim 9 , wherein storing the representation of the modified data field from each iteration in a different delta table is performed in response to the data field already existing in the delta table.
11. The non-transitory computer-readable medium of claim 9 , the operations further comprising:
merging the backup table and the delta tables created in the plurality of iterations after a threshold number of iterations have been completed.
12. The non-transitory computer-readable medium of claim 7 , wherein the backup data for the graph data is a copy of the graph data.
13. A system for analyzing graph data representing relationships among items, the graph data including a plurality of vertices connected by one or more edges representing the relationships among items in the graph, each of the vertices associated with one or more data fields, comprising:
memory for storing data; and
one or more processors operable to perform operations comprising:
identifying requests by a worker system in a distributed computing system including a plurality of workers systems to which the graph data is partitioned, each request to change a data field associated with a particular vertex in the graph data responsive to an operation performed during an analysis of the graph data, the request including a new value for the data field, and for each request:
comparing, by the worker system, the new value with an existing value of the data field associated with the particular vertex in the graph data;
modifying, by the worker system, the data field in the graph data based on the comparison, wherein modifying the graph data does not include modifying backup data for the graph data stored in a backup table; and
responsive to modification of the data field in the graph data, storing, by the worker system in a memory of the worker system, a representation of the modified data field in a respective delta table for the request, the respective delta table different than the backup table;
determining, by the worker system, after each request, whether a number of respective delta tables has reached a merge threshold;
in response to determining that the number of respective delta tables has reached the merge threshold:
merging, by the worker system, the backup table and the delta tables to create a new backup table; and
replacing, by the worker system, the backup table with the new backup table.
14. The system of claim 13 , the operations further comprising:
responsive to an additional modification of the data field in the graph data after storing the representation of the modified data field in the delta table, storing an additional representation of the modified data field in an additional delta table.
15. The system of claim 13 , wherein comparing the new value and modifying the data field are repeated for a plurality of iterations, and the operations further comprising:
storing a representation of the modified data field from each iteration in a different delta table.
16. The system of claim 15 , wherein storing the representation of the modified data field from each iteration in a different delta table is performed in response to the data field already existing in the delta table.
17. The system of claim 15 , the operations further comprising:
merging the backup table and the delta tables created in the plurality of iterations after a threshold number of iterations have been completed.