Table data processing using a change tracking stream
A system includes one or more processors and data storage containing instructions executable by the one or more processors to perform operations. The operations include storing table data in a plurality of partitions of a storage device. Metadata is retrieved from a first partition of the plurality of partitions. The metadata includes a plurality of change tracking entries stored as a change tracking stream. A lineage of modifications made to the table data is determined using the plurality of change tracking entries. A report of one or more transactions performed on the table data is generated. The one or more transactions are included in the lineage of modifications.
1. A method comprising:
storing, by at least one hardware processor, table data in a plurality of partitions of a storage device;
retrieving, by the at least one hardware processor, metadata from a first partition of the plurality of partitions, the metadata comprising a plurality of change tracking entries stored as a change tracking stream;
determining, by the at least one hardware processor, a lineage of modifications made to the table data using the plurality of change tracking entries; and
generating, by the at least one hardware processor, a report of one or more transactions performed on the table data, the one or more transactions included in the lineage of modifications.
2. The method of claim 1 , further comprising:
determining using the lineage of modifications, at least a second partition storing the table data prior to performing the one or more transactions.
3. The method of claim 2 , further comprising:
determining using the lineage of modifications, at least a third partition storing modified data generated after the performing of the one or more transactions on the table data; and
updating the report to indicate the at least a second partition and the at least a third partition.
4. The method of claim 1 , further comprising:
configuring the plurality of partitions as immutable storage partitions associated with the storage device.
5. The method of claim 1 , further comprising:
storing transaction data for the one or more transactions in the change tracking stream, the transaction data comprising one or more of: time of initiating performing the one or more transactions, time of completion of the performing of the one or more transactions, and identification of rows in the table data affected by the performing of the one or more transactions.
6. The method of claim 1 , further comprising:
querying the lineage of modifications to determine a delta indicating a net change between a first timestamp and a second timestamp associated with performing the one or more transactions.
7. The method of claim 6 , further comprising:
querying the lineage of modifications to further determine a change tracking summary indicating changes that occurred on the table data between the first timestamp and the second timestamp.
8. The method of claim 1 , wherein the one or more transactions comprise one or more of an insert, a delete, an update, and a merge of the table data.
9. The method of claim 1 , wherein the one or more transactions comprise a plurality of transactions, and wherein the plurality of change tracking entries comprise one or more sequential change tracking entries corresponding to a subset of the plurality of transactions that have been fully executed on the table data.
10. A system, comprising:
one or more processors; and
data storage containing instructions executable by the one or more processors to perform operations comprising:
storing table data in a plurality of partitions of a storage device;
retrieving metadata from a first partition of the plurality of partitions, the metadata comprising a plurality of change tracking entries stored as a change tracking stream;
determining a lineage of modifications made to the table data using the plurality of change tracking entries; and
generating a report of one or more transactions performed on the table data, the one or more transactions included in the lineage of modifications.
11. The system of claim 10 , the operations further comprising:
determining using the lineage of modifications, at least a second partition storing the table data prior to performing the one or more transactions.
12. The system of claim 11 , the operations further comprising:
determining using the lineage of modifications, at least a third partition storing modified data generated after the performing of the one or more transactions on the table data; and
updating the report to indicate the at least a second partition and the at least a third partition.
13. The system of claim 10 , the operations further comprising:
configuring the plurality of partitions as immutable storage partitions associated with the storage device.
14. The system of claim 10 , the operations further comprising:
storing transaction data for the one or more transactions in the change tracking stream, the transaction data comprising one or more of: time of initiating performing the one or more transactions, time of completion of the performing of the one or more transactions, and identification of rows in the table data affected by the performing of the one or more transactions.
15. The system of claim 10 , the operations further comprising:
querying the lineage of modifications to determine a delta indicating a net change between a first timestamp and a second timestamp associated with performing the one or more transactions.
16. The system of claim 15 , the operations further comprising:
querying the lineage of modifications to further determine a change tracking summary indicating changes that occurred on the table data between the first timestamp and the second timestamp.
17. The system of claim 10 , wherein the one or more transactions comprise one or more of an insert, a delete, an update, and a merge of the table data.
18. The system of claim 10 , wherein the one or more transactions comprise a plurality of transactions, and wherein the plurality of change tracking entries comprise one or more sequential change tracking entries corresponding to a subset of the plurality of transactions that have been fully executed on the table data.
19. A non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
storing table data in a plurality of partitions of a storage device;
retrieving metadata from a first partition of the plurality of partitions, the metadata comprising a plurality of change tracking entries stored as a change tracking stream;
determining a lineage of modifications made to the table data using the plurality of change tracking entries; and
generating a report of one or more transactions performed on the table data, the one or more transactions included in the lineage of modifications.
20. The non-transitory computer-readable storage media of claim 19 , the operations further comprising:
determining using the lineage of modifications, at least a second partition storing the table data prior to performing the one or more transactions.
21. The non-transitory computer-readable storage media of claim 20 , the operations further comprising:
determining using the lineage of modifications, at least a third partition storing modified data generated after the performing of the one or more transactions on the table data; and
updating the report to indicate the at least a second partition and the at least a third partition.
22. The non-transitory computer-readable storage media of claim 19 , the operations further comprising:
configuring the plurality of partitions as immutable storage partitions associated with the storage device.
23. The non-transitory computer-readable storage media of claim 19 , the operations further comprising:
storing transaction data for the one or more transactions in the change tracking stream, the transaction data comprising one or more of: time of initiating performing the one or more transactions, time of completion of the performing of the one or more transactions, and identification of rows in the table data affected by the performing of the one or more transactions.
24. The non-transitory computer-readable storage media of claim 19 , the operations further comprising:
querying the lineage of modifications to determine a delta indicating a net change between a first timestamp and a second timestamp associated with performing the one or more transactions.
25. The non-transitory computer-readable storage media of claim 24 , the operations further comprising:
querying the lineage of modifications to further determine a change tracking summary indicating changes that occurred on the table data between the first timestamp and the second timestamp.
26. The non-transitory computer-readable storage media of claim 19 , wherein the one or more transactions comprise one or more of an insert, a delete, an update, and a merge of the table data.
27. The non-transitory computer-readable storage media of claim 19 , wherein the one or more transactions comprise a plurality of transactions, and wherein the plurality of change tracking entries comprise one or more sequential change tracking entries corresponding to a subset of the plurality of transactions that have been fully executed on the table data.