Scheduling data processing tasks using a stream of tracking entries
Systems, methods, and devices for scheduling of data processing tasks are disclosed. A method includes performing a modification on a first set of immutable partitions storing database data to generate a second set of immutable partitions. The second set is associated with a modified version of the database data. A change tracking entry is entered in a stream of tracking entries based on committing the modification. The change tracking entry includes an indication of the modification on the first set of immutable partitions. A stream offset of the stream of tracking entries is advanced based on the entering of the change tracking entry in the stream of tracking entries. The stream offset indicates a timestamp associated with a latest committed modification to the database data. A data processing task is scheduled for execution on the modified version of the database data based on the advancing of the stream offset.
1. A method comprising:
performing, by at least one hardware processor, a modification on a first set of immutable partitions storing database data to generate a second set of immutable partitions, the second set of immutable partitions associated with a modified version of the database data;
entering a change tracking entry in a stream of tracking entries based on committing the modification, the change tracking entry comprising an indication of the modification on the first set of immutable partitions;
advancing a stream offset of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the stream offset indicating a timestamp associated with a latest committed modification to the database data; and
scheduling a data processing task to be executed on the modified version of the database data based on the advancing of the stream offset.
2. The method of claim 1 , further comprising:
configuring the first set of immutable partitions as a first table; and
configuring the second set of immutable partitions as a second table, the second table being non-overlapping with the first table.
3. The method of claim 1 , further comprising:
in response to a trigger event including the advancing of the stream offset, automatically executing the data processing task on the modified version of the database data; and
generating a task history entry indicating the data processing task was successfully executed.
4. The method of claim 3 , further comprising:
executing a query on the stream of tracking entries to determine a comprehensive change tracking summary between the timestamp and a second timestamp.
5. The method of claim 4 , wherein the comprehensive change tracking summary indicates intermediate modifications that occurred on the first set of immutable partitions between the timestamp and the second timestamp.
6. The method of claim 3 , wherein the trigger event further comprises a passing of a predefined time period.
7. The method of claim 3 , wherein the trigger event further comprises a threshold number of modifications made to the first set of immutable partitions.
8. The method of claim 1 , wherein advancing the stream offset further comprises:
advancing a retention boundary of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the retention boundary indicating an earliest timestamp the stream offset is associated with.
9. The method of claim 8 , further comprising:
configuring the retention boundary based on a retention period associated with the first set of immutable partitions.
10. The method of claim 8 , further comprising:
executing a query on the stream of tracking entries to determine a delta between at least two timestamps associated with committed modification to the database data, based on reading a changeset between the stream offset and the retention boundary.
11. A system comprising:
at least one processor; and
one or more non-transitory computer readable storage media containing instructions executable by the at least one processor for causing the at least one processor to perform operations comprising:
performing a modification on a first set of immutable partitions storing database data to generate a second set of immutable partitions, the second set of immutable partitions associated with a modified version of the database data;
entering a change tracking entry in a stream of tracking entries based on committing the modification, the change tracking entry comprising an indication of the modification on the first set of immutable partitions;
advancing a stream offset of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the stream offset indicating a timestamp associated with a latest committed modification to the database data; and
scheduling a data processing task to be executed on the modified version of the database data based on the advancing of the stream offset.
12. The system of claim 11 , the operations further comprising:
configuring the first set of immutable partitions as a first table; and
configuring the second set of immutable partitions as a second table, the second table being non-overlapping with the first table.
13. The system of claim 11 , the operations further comprising:
in response to a trigger event including the advancing of the stream offset, automatically executing the data processing task on the modified version of the database data; and
generating a task history entry indicating the data processing task was successfully executed.
14. The system of claim 13 , the operations further comprising:
executing a query on the stream of tracking entries to determine a comprehensive change tracking summary between the timestamp and a second timestamp.
15. The system of claim 14 , wherein the comprehensive change tracking summary indicates intermediate modifications that occurred on the first set of immutable partitions between the timestamp and the second timestamp.
16. The system of claim 13 , wherein the trigger event further comprises a passing of a predefined time period.
17. The system of claim 13 , wherein the trigger event further comprises a threshold number of modifications made to the first set of immutable partitions.
18. The system of claim 11 , wherein the operations for advancing the stream offset further comprise:
advancing a retention boundary of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the retention boundary indicating an earliest timestamp the stream offset is associated with.
19. The system of claim 18 , the operations further comprising:
configuring the retention boundary based on a retention period associated with the first set of immutable partitions.
20. The system of claim 18 , the operations further comprising:
executing a query on the stream of tracking entries to determine a delta between at least two timestamps associated with committed modification to the database data, based on reading a changeset between the stream offset and the retention boundary.
21. One or more non-transitory computer readable storage media containing instructions executable by at least one processor for causing the at least one processor to perform operations comprising:
performing a modification on a first set of immutable partitions storing database data to generate a second set of immutable partitions, the second set of immutable partitions associated with a modified version of the database data;
entering a change tracking entry in a stream of tracking entries based on committing the modification, the change tracking entry comprising an indication of the modification on the first set of immutable partitions;
advancing a stream offset of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the stream offset indicating a timestamp associated with a latest committed modification to the database data; and
scheduling a data processing task to be executed on the modified version of the database data based on the advancing of the stream offset.
22. The non-transitory computer readable storage media of claim 21 , the operations further comprising:
configuring the first set of immutable partitions as a first table; and
configuring the second set of immutable partitions as a second table, the second table being non-overlapping with the first table.
23. The non-transitory computer readable storage media of claim 21 , the operations further comprising:
in response to a trigger event including the advancing of the stream offset, automatically executing the data processing task on the modified version of the database data; and
generating a task history entry indicating the data processing task was successfully executed.
24. The non-transitory computer readable storage media of claim 23 , the operations further comprising:
executing a query on the stream of tracking entries to determine a comprehensive change tracking summary between the timestamp and a second timestamp.
25. The non-transitory computer readable storage media of claim 11 , wherein the operations for advancing the stream offset further comprise:
advancing a retention boundary of the stream of tracking entries based on the entering of the change tracking entry in the stream of tracking entries, the retention boundary indicating an earliest timestamp the stream offset is associated with.
26. The non-transitory computer readable storage media of claim 25 , the operations further comprising:
configuring the retention boundary based on a retention period associated with the first set of immutable partitions.
27. The non-transitory computer readable storage media of claim 25 , the operations further comprising:
executing a query on the stream of tracking entries to determine a delta between at least two timestamps associated with committed modification to the database data, based on reading a changeset between the stream offset and the retention boundary.