Task scheduling in database systems

1. A method comprising:

executing a transaction on a first table, the first table comprising database data, the executing of the transaction comprising generating a second table corresponding to a new version of the first table, the second table being a separate table from the first table;

after the transaction is fully executed based at least in part on the second table being generated as the separate table, generating a change tracking entry, the change tracking entry comprising an indication of at least one modification made to the first table by the executing of the transaction;

advancing a change tracking stream at least in part by entering the generated change tracking entry into the change tracking stream, the advancing of the change tracking stream comprising advancing a stream offset and a retention boundary in the change tracking stream, the stream offset corresponding to a first position in the change tracking stream and the retention boundary corresponding to a second position in the change tracking stream, the change tracking stream including a series of committed transactions; and

in response to a trigger event including the advancing of the change tracking stream, automatically executing a pre-defined task on the second table and generating a task history entry indicating the task was successfully executed.

2. The method of claim 1 , wherein:

executing the transaction on the first table comprises ingesting new data into the first table; and

the method further comprises:

storing the new data in a staging table; and

transforming the new data in the staging table for storage in one or more target tables.

3. The method of claim 1 , wherein executing the transaction on the first table comprises one or more of inserting data into the first table, deleting data from the first table, updating data in the first table, and merging data in the first table.

4. The method of claim 1 , wherein the change tracking entry further comprises a transaction timestamp associated with the executing of the transaction, the change tracking stream being ordered chronologically according to transaction timestamps.

5. The method of claim 4 , wherein the indication of at least one modification made to the first table by the executing of the transaction indicates:

any one or more micro-partitions that were added to the first table by the executing of the transaction;

any one or more micro-partitions that were removed from the first table by the executing of the transaction;

any one or more rows that were added to the first table by the executing of the transaction; and

any one or more rows that were removed from the first table by the executing of the transaction.

6. The method of claim 5 , further comprising reading the change tracking stream to determine a delta for the first table between a first transaction timestamp and a second transaction timestamp.

7. The method of claim 5 , further comprising executing a query on the change tracking stream to determine a comprehensive change tracking summary between a first transaction timestamp and a second transaction timestamp, the comprehensive change tracking summary indicating all intermediate modifications that occurred on the first table between the first transaction timestamp and the second transaction timestamp.

8. The method of claim 1 , wherein the trigger event further comprises a passing of a predefined time period.

9. The method of claim 1 , wherein the trigger event further comprises a threshold number of modifications having been made to the first table.

10. A system comprising:

at least one processor; and

one or more non-transitory computer readable storage media containing instructions executable by the at least one processor for causing the at least one processor to perform operations comprising:

executing a transaction on a first table, the first table comprising database data, the executing of the transaction comprising generating a second table corresponding to a new version of the first table, the second table being a separate table from the first table;

after the transaction is fully executed based at least in part on the second table being generated as the separate table, generating a change tracking entry, the change tracking entry comprising an indication of at least one modification made to the first table by the executing of the transaction;

advancing a change tracking stream at least in part by entering the generated change tracking entry into the change tracking stream, the advancing of the change tracking stream comprising advancing a stream offset and a retention boundary in the change tracking stream, the stream offset corresponding to a first position in the change tracking stream and the retention boundary corresponding to a second position in the change tracking stream, the change tracking stream including a series of committed transactions; and

in response to a trigger event including advancing of the change tracking stream, automatically executing a pre-defined task on the second table and generating a task history entry indicating the task was successfully executed.

11. The system of claim 10 , wherein:

executing the transaction on the first table comprises ingesting new data into the first table; and

the operations further comprise:

storing the new data in a staging table; and

transforming the new data in the staging table for storage in one or more target tables.

12. The system of claim 10 , wherein executing the transaction on the first table comprises one or more of inserting data into the first table, deleting data from the first table, updating data in the first table, and merging data in the first table.

13. The system of claim 10 , wherein the change tracking entry further comprises a transaction timestamp associated with the executing of the transaction, the change tracking stream being ordered chronologically according to transaction timestamps.

14. The system of claim 13 , wherein the indication of one or more modifications made to the first table by the executing of the transaction indicates:

any one or more micro-partitions that were added to the first table by the executing of the transaction;

any one or more micro-partitions that were removed from the first table by the executing of the transaction;

any one or more rows that were added to the first table by the executing of the transaction; and

any one or more rows that were removed from the first table by the executing of the transaction.

15. The system of claim 14 , the operations further comprising reading the change tracking stream to determine a delta for the first table between a first transaction timestamp and a second transaction timestamp.

16. The system of claim 14 , the operations further comprising executing a query on the change tracking stream to determine a comprehensive change tracking summary between a first transaction timestamp and a second transaction timestamp, the comprehensive change tracking summary indicating all intermediate modifications that occurred on the first table between the first transaction timestamp and the second transaction timestamp.

17. The system of claim 10 , wherein advancing the change tracking stream comprises advancing a stream offset in the change tracking stream.

18. The system of claim 10 , wherein the trigger event further comprises a passing of a predefined time period.

19. The system of claim 10 , wherein the trigger event further comprises a threshold number of modifications having been made to the first table.

20. One or more non-transitory computer readable storage media containing instructions executable by at least one processor for causing the at least one processor to perform operations comprising:

executing a transaction on a first table, the first table comprising database data, the executing of the transaction comprising generating a second table corresponding to a new version of the first table, the second table being a separate table from the first table;

after the transaction is fully executed based at least in part on the second table being generated as the separate table, generating a change tracking entry, the change tracking entry comprising an indication of at least one modification made to the first table by the executing of the transaction;

advancing a change tracking stream at least in part by entering the generated change tracking entry into the change tracking stream, the advancing of the change tracking stream comprising advancing a stream offset and a retention boundary in the change tracking stream, the stream offset corresponding to a first position in the change tracking stream and the retention boundary corresponding to a second position in the change tracking stream, the change tracking stream including a series of committed transactions; and

in response to a trigger event including advancing of the change tracking stream, automatically executing a pre-defined task on the second table and generating a task history entry indicating the task was successfully executed.

21. The non-transitory computer readable storage media of claim 20 , wherein:

executing the transaction on the first table comprises ingesting new data into the first table; and

the operations further comprise:

storing the new data in a staging table; and

transforming the new data in the staging table for storage in one or more target tables.

22. The non-transitory computer readable storage media of claim 20 , wherein executing the transaction on the first table comprises one or more of inserting data into the first table, deleting data from the first table, updating data in the first table, and merging data in the first table.

23. The non-transitory computer readable storage media of claim 20 , wherein the change tracking entry further comprises a transaction timestamp associated with the executing of the transaction, the change tracking stream being ordered chronologically according to transaction timestamps.

24. The non-transitory computer readable storage media of claim 23 , wherein the indication of one or more modifications made to the first table by the executing of the transaction indicates:

any one or more micro-partitions that were added to the first table by the executing of the transaction;

any one or more micro-partitions that were removed from the first table by the executing of the transaction;

any one or more rows that were added to the first table by the executing of the transaction; and

any one or more rows that were removed from the first table by the executing of the transaction.

25. The non-transitory computer readable storage media of claim 24 , the operations further comprising reading the change tracking stream to determine a delta for the first table between a first transaction timestamp and a second transaction timestamp.

26. The non-transitory computer readable storage media of claim 24 , the operations further comprising executing a query on the change tracking stream to determine a comprehensive change tracking summary between a first transaction timestamp and a second transaction timestamp, the comprehensive change tracking summary indicating all intermediate modifications that occurred on the first table between the first transaction timestamp and the second transaction timestamp.

27. The non-transitory computer readable storage media of claim 20 , wherein advancing the change tracking stream comprises advancing a stream offset in the change tracking stream.

28. The non-transitory computer readable storage media of claim 20 , wherein the trigger event further comprises a passing of a predefined time period.

29. The non-transitory computer readable storage media of claim 20 , wherein the trigger event further comprises a threshold number of modifications having been made to the first table.