Execution and consistency model for materialized tables

1. A system comprising:

at least one hardware processor; and

at least one memory storing instructions that cause the at least one hardware processor to perform operations comprising:

determining a dependency graph including a plurality of nodes coupled with edges, the dependency graph comprising at least a first node of the plurality of nodes with a first materialized table (MT) and a second node of the plurality of nodes with a second MT from which the first MT depends, the first node with the first MT comprising a query on one or more base tables and a lag duration value, the query defines the first MT and the lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance before a subsequent refresh is initiated;

selecting a tick period for a set of ticks based on the lag duration value, the set of ticks corresponding to a set of aligned time instances;

scheduling refresh operations for the first MT and the second MT at corresponding time instances from the set of aligned time instances that are within the maximum time period indicated by the lag duration, the corresponding time instances separated by the tick period; and

determining the set of aligned time instances based on a frequency level value and a refresh instance value, the frequency level value being an exponent of two, and the refresh instance value indicating a number of prior ticks that have occurred up to a current time instance.

2. The system of claim 1 , wherein the operations further comprise:

determining a duration of the refresh operations for the first MT and the second MT, the refresh operations initiated at one of the corresponding time instances.

3. The system of claim 2 , wherein the operations further comprise:

adjusting the tick period based on the duration.

4. The system of claim 3 , wherein adjusting the tick period further comprises:

reducing the tick period below the lag duration value based on the duration of the refresh operations being greater than a pre-configured duration.

5. The system of claim 1 , wherein the operations further comprise:

generating a data definition language (DDL) log of dependencies among the first MT and the second MT; and

applying a graph rendering process to the DDL log of dependencies to generate the dependency graph, the dependency graph comprising a plurality of nodes corresponding to the first MT and the second MT.

6. The system of claim 5 , wherein the operations further comprise:

separating the plurality of nodes into subsets of nodes based on data processing account association, wherein nodes in a subset of the subsets are associated with a common time instance of the set of aligned time instances.

7. The system of claim 6 , wherein the operations further comprise:

configuring processing pipelines based on the set of aligned time instances, each processing pipeline of the processing pipelines corresponding to the nodes associated with the common time instance.

8. The system of claim 7 , wherein the operations further comprise:

selecting a processing pipeline of the processing pipelines based on the corresponding time instances from the set of aligned time instances; and

scheduling the refresh operations for the first MT and the second MT using the processing pipeline.

9. A method comprising:

determining, by at least one hardware processor, a dependency graph including a plurality of nodes coupled with edges, the dependency graph comprising at least a first node of the plurality of nodes with a first materialized table (MT) and a second node of the plurality of nodes with a second MT from which the first MT depends, the first node with the first MT comprising a query on one or more base tables and a lag duration value, the query defines the first MT and the lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance before a subsequent refresh is initiated;

selecting, by the at least one hardware processor, a tick period for a set of ticks based on the lag duration value, the set of ticks corresponding to a set of aligned time instances;

scheduling, by the at least one hardware processor, refresh operations for the first MT and the second MT at corresponding time instances from the set of aligned time instances that are within the maximum time period indicated by the lag duration, the corresponding time instances separated by the tick period; and

determining, by the at least one hardware processor, the set of aligned time instances based on a frequency level value and a refresh instance value, the frequency level value being an exponent of two, and the refresh instance value indicating a number of prior ticks that have occurred up to a current time instance.

10. The method of claim 9 , further comprising:

determining a duration of the refresh operations for the first MT and the second MT, the refresh operations initiated at one of the corresponding time instances.

11. The method of claim 10 , further comprising:

adjusting the tick period based on the duration.

12. The method of claim 11 , wherein the adjusting of the tick period further comprises:

reducing the tick period below the lag duration value based on the duration of the refresh operations being greater than a pre-configured duration.

13. The method of claim 9 , further comprising:

generating a data definition language (DDL) log of dependencies among the first MT and the second MT; and

applying a graph rendering process to the DDL log of dependencies to generate the dependency graph, the dependency graph comprising a plurality of nodes corresponding to the first MT and the second MT.

14. The method of claim 13 , further comprising:

separating the plurality of nodes into subsets of nodes based on data processing account association, wherein nodes in a subset of the subsets are associated with a common time instance of the set of aligned time instances.

15. The method of claim 14 , further comprising:

configuring processing pipelines based on the set of aligned time instances, each processing pipeline of the processing pipelines corresponding to the nodes associated with the common time instance.

16. The method of claim 15 , further comprising:

selecting a processing pipeline of the processing pipelines based on the corresponding time instances from the set of aligned time instances; and

scheduling the refresh operations for the first MT and the second MT using the processing pipeline.

17. A computer-storage medium comprising instructions that, when executed by one or more processors of a machine, configure the machine to perform operations comprising:

determining a dependency graph including a plurality of nodes coupled with edges, the dependency graph comprising at least a first node of the plurality of nodes with a first materialized table (MT) and a second node of the plurality of nodes with a second MT from which the first MT depends, the first node with the first MT comprising a query on one or more base tables and a lag duration value, the query defines the first MT and the lag duration value indicates a maximum time period that a result of a prior refresh of the query can lag behind a current time instance before a subsequent refresh is initiated;

selecting a tick period for a set of ticks based on the lag duration value, the set of ticks corresponding to a set of aligned time instances;

scheduling refresh operations for the first MT and the second MT at corresponding time instances from the set of aligned time instances that are within the maximum time period indicated by the lag duration, the corresponding time instances separated by the tick period; and

determining, by the at least one hardware processor, the set of aligned time instances based on a frequency level value and a refresh instance value, the frequency level value being an exponent of two, and the refresh instance value indicating a number of prior ticks that have occurred up to a current time instance.

18. The computer-storage medium of claim 17 , the operations further comprising:

determining a duration of the refresh operations for the first MT and the second MT, the refresh operations initiated at one of the corresponding time instances.

19. The computer-storage medium of claim 18 , the operations further comprising:

adjusting the tick period based on the duration.

20. The computer-storage medium of claim 19 , wherein the adjusting of the tick period further comprises:

reducing the tick period below the lag duration value based on the duration of the refresh operations being greater than a pre-configured duration.

21. The computer-storage medium of claim 17 , the operations further comprising:

generating a data definition language (DDL) log of dependencies among the first MT and the second MT; and

applying a graph rendering process to the DDL log of dependencies to generate the dependency graph, the dependency graph comprising a plurality of nodes corresponding to the first MT and the second MT.

22. The computer-storage medium of claim 21 , the operations further comprising:

separating the plurality of nodes into subsets of nodes based on data processing account association, wherein nodes in a subset of the subsets are associated with a common time instance of the set of aligned time instances.

23. The computer-storage medium of claim 22 , the operations further comprising:

configuring processing pipelines based on the set of aligned time instances, each processing pipeline of the processing pipelines corresponding to the nodes associated with the common time instance.

24. The computer-storage medium of claim 23 , the operations further comprising:

selecting a processing pipeline of the processing pipelines based on the corresponding time instances from the set of aligned time instances; and

scheduling the refresh operations for the first MT and the second MT using the processing pipeline.