Query refresh using multiple processing pipelines
A system includes at least one hardware processor and at least one memory storing instructions that cause the at least one hardware processor to perform operations. The operations include generating a log of changes posted to a plurality of intermediate materialized tables (MTs) during execution of a query in a network-based database system. The query is associated with a source MT that the intermediate MTs depend on. The operations include rendering the log of changes into a dependency graph. The operations include configuring a plurality of processing pipelines based on the dependency graph. The operations include performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation. The refreshes are performed responsive to detecting an instruction for a refresh operation on the source MT.
1 . A system comprising:
at least one hardware processor; and
at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:
generating a source materialized table (MT) based on a table definition, the table definition comprising a query on one or more base tables, and a lag duration value, the lag duration value indicating a maximum time period that a result of a prior refresh of the query can lag behind a current time instance;
generating a log of changes posted to a plurality of intermediate MTs during execution of the query in a network-based database system, the query associated with the source MT that the intermediate MTs depend on;
rendering the log of changes into a dependency graph;
configuring a plurality of processing pipelines based on the dependency graph; and
responsive to detecting an instruction for a refresh operation on the source MT, performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation.
2 . The system of claim 1 , wherein the operations comprise:
partitioning the dependency graph into a plurality of sub-graphs based on an association of the changes to one or more user accounts of the network-based database system.
3 . The system of claim 2 , wherein the operations comprise:
configuring the plurality of processing pipelines to correspond to the plurality of sub-graphs.
4 . The system of claim 1 , wherein the operations comprise:
configuring a sub-graph of the plurality of sub-graphs to indicate dependencies among a subset of the plurality of intermediate MTs.
5 . The system of claim 4 , wherein the operations comprise:
configuring the subset with at least one composite MT, the at least one composite MT depending on at least two remaining MTs of the subset.
6 . The system of claim 1 , wherein the operations comprise:
performing the refreshes on the plurality of intermediate MTs in parallel in the plurality of processing pipelines to complete the refresh operation.
7 . The system of claim 1 , wherein the operations comprise:
generating the plurality of intermediate MTs based on a plurality of data processing operations in a query plan of the query.
8 . A method comprising:
generating, by at least one hardware processor, a source materialized table (MT) based on a table definition, the table definition comprising a query on one or more base tables, and a lag duration value, the lag duration value indicating a maximum time period that a result of a prior refresh of the query can lag behind a current time instance;
generating, by the at least one hardware processor, a log of changes posted to a plurality of intermediate MTs during execution of the query in a network-based database system, the query associated with the source MT that the intermediate MTs depend on;
rendering the log of changes into a dependency graph;
configuring a plurality of processing pipelines based on the dependency graph; and
responsive to detecting an instruction for a refresh operation on the source MT, performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation.
9 . The method of claim 8 , further comprising:
partitioning the dependency graph into a plurality of sub-graphs based on an association of the changes to one or more user accounts of the network-based database system.
10 . The method of claim 9 , further comprising:
configuring the plurality of processing pipelines to correspond to the plurality of sub-graphs.
11 . The method of claim 9 , further comprising:
configuring a sub-graph of the plurality of sub-graphs to indicate dependencies among a subset of the plurality of intermediate MTs.
12 . The method of claim 11 , further comprising:
configuring the subset with at least one composite MT, the at least one composite MT depending on at least two remaining MTs of the subset.
13 . The method of claim 8 , further comprising:
performing the refreshes on the plurality of intermediate MTs in parallel in the plurality of processing pipelines to complete the refresh operation.
14 . The method of claim 8 , further comprising:
generating the plurality of intermediate MTs based on a plurality of data processing operations in a query plan of the query.
15 . A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:
generating a source materialized table (MT) based on a table definition, the table definition comprising a query on one or more base tables, and a lag duration value, the lag duration value indicating a maximum time period that a result of a prior refresh of the query can lag behind a current time instance;
generating a log of changes posted to a plurality of intermediate MTs during execution of the query in a network-based database system, the query associated with the source MT that the intermediate MTs depend on;
rendering the log of changes into a dependency graph;
configuring a plurality of processing pipelines based on the dependency graph; and
responsive to detecting an instruction for a refresh operation on the source MT, performing refreshes on one or more of the plurality of intermediate MTs in at least one of the plurality of processing pipelines to complete the refresh operation.
16 . The computer-storage medium of claim 15 , the operations further comprising:
partitioning the dependency graph into a plurality of sub-graphs based on an association of the changes to one or more user accounts of the network-based database system.
17 . The computer-storage medium of claim 16 , the operations further comprising:
configuring the plurality of processing pipelines to correspond to the plurality of sub-graphs.
18 . The computer-storage medium of claim 16 , the operations further comprising:
configuring a sub-graph of the plurality of sub-graphs to indicate dependencies among a subset of the plurality of intermediate MTs.
19 . The computer-storage medium of claim 18 , the operations further comprising:
configuring the subset with at least one composite MT, the at least one composite MT depending on at least two remaining MTs of the subset.
20 . The computer-storage medium of claim 15 , the operations further comprising:
performing the refreshes on the plurality of intermediate MTs in parallel in the plurality of processing pipelines to complete the refresh operation.
21 . The computer-storage medium of claim 15 , the operations further comprising:
generating the plurality of intermediate MTs based on a plurality of data processing operations in a query plan of the query.