Computer data system data source refreshing using an update propagation graph
Described are methods, systems and computer readable media for data source refreshing.
1. A system for updating a data object using an update propagation graph, the system comprising:
one or more hardware processors coupled to a nontransitory computer readable medium having stored thereon software instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving, at a remote query processor, a first query from a client, the first query referencing a plurality of data objects including one or more data sources;
determining dependencies between the data objects in the first query;
generating an update propagation graph (UPG) in a context of the remote query processor based on the first query, the UPG having a plurality of nodes each corresponding to one of the data objects, the UPG having a structure representing the dependencies between the data objects in the first query;
determining a logical clock has transitioned to an updating state;
processing one or more changes to one or more corresponding data sources, wherein the changes are processed according to an order determined by the UPG, wherein change processing for each data source includes:
invoking a data source refresh method for a data source for which changes are being processed;
determining whether a priority queue for the data source is empty;
when the priority queue is not empty, retrieving a next change notification message from the priority queue and delivering the change notification to a corresponding data source and repeating determining whether the priority is queue is empty; and
when the priority queue is empty, setting the logical clock to an idle state;
updating the structure of the UPG in response to a second query being executed by the remote query processor; and
after the updating, processing another one or more changes according to a different order determined by the updated UPG.
2. The system of claim 1 , wherein the change notifications can include one of a data add notification, a data modify notification, a data delete notification and a data reindex notification.
3. The system of claim 1 , wherein each node of the UPG includes one of a dynamic node, a static node and an internal node.
4. The system of claim 1 , wherein the operations further include adding an additional notification to the priority queue from a listener associated with the UPG.
5. The system of claim 1 , wherein one or more nodes represents a table data object.
6. The system of claim 1 , wherein one or more nodes represents a nontabular data object.
7. The system of claim 1 , wherein a node can include an internal node that is a data source for another internal node or a data source.
8. The system of claim 1 , wherein the operations further comprise determining that a data source has been garbage collected and not performing updating processing for the garbage collected data source.
9. The system of claim 1 , wherein the operations further include:
constructing the plurality of nodes in a construction order based on the dependencies between the data objects in the first query,
wherein the order in which the data sources are processed is the construction order.
10. A method for updating a data object using an update propagation graph, the method comprising:
receiving, at a remote query processor, a first query from a client, the first query referencing a plurality of data objects including one or more data sources;
determining dependencies between the data objects in the first query;
generating an update propagation graph (UPG) in a context of the remote query processor based on the first query, the UPG having a plurality of nodes each corresponding to one of the data objects, the UPG having a structure representing the dependencies between the data objects in the first query;
determining a logical clock has transitioned to an updating state;
processing one or more changes to one or more corresponding data sources, wherein the changes are processed according to an order determined by the UPG, wherein change processing for each data source includes:
invoking a data source refresh method for a data source for which changes are being processed;
determining whether a priority queue for the data source is empty;
when the priority queue is not empty, retrieving a next change notification message from the priority queue and delivering the change notification to a corresponding data source and repeating determining whether the priority is queue is empty; and
when the priority queue is empty, setting the logical clock to an idle state;
updating the structure of the UPG in response to a second query being executed by the remote query processor; and
after the updating, processing another one or more changes according to a different order determined by the updated UPG.
11. The method of claim 10 , wherein the change notifications can include one of a data add notification, a data modify notification, a data delete notification and a data reindex notification.
12. The method of claim 10 , wherein each node of the UPG includes one of a dynamic node, a static node and an internal node.
13. The method of claim 10 , further comprising adding an additional notification to the priority queue from a listener associated with the UPG.
14. The method of claim 10 , wherein one or more nodes represents a table data object.
15. The method of claim 10 , wherein a node can include an internal node that is a data source for another internal node or a data source.
16. The method of claim 10 , further comprising determining that a data source has been garbage collected and not performing updating processing for the garbage collected data source.
17. The method of claim 10 , further comprising:
constructing the plurality of nodes in a construction order based on the dependencies between the data objects in the first query,
wherein the order in which the data sources are processed is the construction order.
18. A nontransitory computer readable medium having stored thereon software instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving, at a remote query processor, a first query from a client, the first query referencing a plurality of data objects including one or more data sources;
determining dependencies between the data objects in the first query;
generating an update propagation graph (UPG) in a context of the remote query processor based on the first query, the UPG having a plurality of nodes each corresponding to one of the data objects, the UPG having a structure representing the dependencies between the data objects in the first query;
determining a logical clock has transitioned to an updating state;
processing one or more changes to one or more corresponding data sources, wherein the changes are processed according to an order determined by the UPG, wherein change processing for each data source includes:
invoking a data source refresh method for a data source for which changes are being processed;
determining whether a priority queue for the data source is empty;
when the priority queue is not empty, retrieving a next change notification message from the priority queue and delivering the change notification to a corresponding data source and repeating determining whether the priority is queue is empty; and
when the priority queue is empty, setting the logical clock to an idle state;
updating the structure of the UPG in response to a second query being executed by the remote query processor; and
after the updating, processing another one or more changes according to a different order determined by the updated UPG.
19. The nontransitory computer readable medium of claim 18 , wherein the operations further include:
constructing the plurality of nodes in a construction order based on the dependencies between the data objects in the first query,
wherein the order in which the data sources are processed is the construction order.