Tracking intermediate changes in database data
Systems, methods, and devices for tracking a series of changes to database data are disclosed. A method includes executing a transaction to modify data in a micro-partition of a table of a database by generating a new micro-partition that embodies the transaction. The method includes associating transaction data with the new micro-partition, wherein the transaction data comprises a timestamp when the transaction was fully executed, and further includes associating modification data with the new micro-partition that comprises an indication of one or more rows of the table that were modified by the transaction. The method includes joining the transaction data with the modification data to generate joined data and querying the joined data to determine a listing of intermediate modifications made to the table between a first timestamp and a second timestamp.
1. A method comprising:
storing data in a first micro-partition of a table, the first micro-partition including a plurality of columns individually encrypted;
receiving a command related to the stored data in the first micro-partition;
decrypting one or more columns of the first micro-partition to execute a transaction to generate transaction data in response to receiving the command;
performing one or more modifications to the decrypted one or more columns in the first micro-partition to generate a second micro-partition, the second micro-partition including a plurality of columns individually encrypted;
removing the first micro-partition from the table;
generating a delta table, the delta table storing final changes to the stored data in the second micro-partition from the first micro-partition, the delta table being encrypted using a first file key; and
generating modification data, the modification data tracking the modifications performed to the stored data between the second micro-partition and the first micro-partition, the modification data being encrypted using a second file key.
2. The method of claim 1 , further comprising:
joining the modification data with the transaction data to generate joined data, the joined data indicating intermediate changes to the stored data between the second micro-partition and the first micro-partition.
3. The method of claim 2 , wherein the transaction data includes one or more of: an identity of an account that initiated the transaction, a first timestamp when the transaction was requested, a second timestamp when execution of the transaction began, a listing of all rows that were modified by the transaction, and details of the modifications.
4. The method of claim 2 , wherein multiple modifications to a row of a plurality of rows in the first micro-partition are performed and the joined data indicates at least one intermediate change to the row between initial and final values.
5. The method of claim 1 , further comprising:
determining tuple changes between each of a series of sequential micro-partition pairs between the first and second micro-partitions.
6. The method of claim 1 , wherein the modification data is stored in the second micro-partition as metadata, wherein the modification data includes a lineage of modifications made to the table.
7. The method of claim 1 , further comprising:
wherein the delta table includes information indicating which rows have been modified, final values of rows that have been modified between the first and second micro-partitions, and an action type of the modification for each row modified.
8. A system comprising:
one or more processors of a machine; and
a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
storing data in a first micro-partition of a table, the first micro-partition including a plurality of columns individually encrypted;
receiving a command related to the stored data in the first micro-partition;
decrypting one or more columns of the first micro-partition to execute a transaction to generate transaction data in response to receiving the command;
performing one or more modifications to the decrypted one or more columns in the first micro-partition to generate a second micro-partition, the second micro-partition including a plurality of columns individually encrypted;
removing the first micro-partition from the table;
generating a delta table, the delta table storing final changes to the stored data in the second micro-partition from the first micro-partition, the delta table being encrypted using a first file key; and
generating modification data, the modification data tracking the modifications performed to the stored data between the second micro-partition and the first micro-partition, the modification data being encrypted using a second file key.
9. The system of claim 8 , the operations further comprising:
joining the modification data with the transaction data to generate joined data, the joined data indicating intermediate changes to the stored data between the second micro-partition and the first micro-partition.
10. The system of claim 9 , wherein the transaction data includes one or more of: an identity of an account that initiated the transaction, a first timestamp when the transaction was requested, a second timestamp when execution of the transaction began, a listing of all rows that were modified by the transaction, and details of the modifications.
11. The system of claim 9 , wherein multiple modifications to a row of a plurality of rows in the first micro-partition are performed and the joined data indicates at least one intermediate change to the row between initial and final values.
12. The system of claim 8 , the operations further comprising:
determining tuple changes between each of a series of sequential micro-partition pairs between the first and second micro-partitions.
13. The system of claim 8 , wherein the modification data is stored in the second micro-partition as metadata, wherein the modification data includes a lineage of modifications made to the table.
14. The system of claim 8 , the operations further comprising:
wherein the delta table includes information indicating which rows have been modified, final values of rows that have been modified between the first and second micro-partitions, and an action type of the modification for each row modified.
15. A non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
storing data in a first micro-partition of a table, the first micro-partition including a plurality of columns individually encrypted;
receiving a command related to the stored data in the first micro-partition;
decrypting one or more columns of the first micro-partition to execute a transaction to generate transaction data in response to receiving the command;
performing one or more modifications to the decrypted one or more columns in the first micro-partition to generate a second micro-partition, the second micro-partition including a plurality of columns individually encrypted;
removing the first micro-partition from the table;
generating a delta table, the delta table storing final changes to the stored data in the second micro-partition from the first micro-partition, the delta table being encrypted using a first file key; and
generating modification data, the modification data tracking the modifications performed to the stored data between the second micro-partition and the first micro-partition, the modification data being encrypted using a second file key.
16. The non-transitory computer readable storage media of claim 15 , further comprising:
joining the modification data with the transaction data to generate joined data, the joined data indicating intermediate changes to the stored data between the second micro-partition and the first micro-partition.
17. The non-transitory computer readable storage media of claim 16 , wherein the transaction data includes one or more of: an identity of an account that initiated the transaction, a first timestamp when the transaction was requested, a second timestamp when execution of the transaction began, a listing of all rows that were modified by the transaction, and details of the modifications.
18. The non-transitory computer readable storage media of claim 16 , wherein multiple modifications to a row of a plurality of rows in the first micro-partition are performed and the joined data indicates at least one intermediate change to the row between initial and final values.
19. The non-transitory computer readable storage media of claim 15 , further comprising:
determining tuple changes between each of a series of sequential micro-partition pairs between the first and second micro-partitions.
20. The non-transitory computer readable storage media of claim 15 , wherein the modification data is stored in the second micro-partition as metadata, wherein the modification data includes a lineage of modifications made to the table.
21. The non-transitory computer readable storage media of claim 15 , further comprising:
wherein the delta table includes information indicating which rows have been modified, final values of rows that have been modified between the first and second micro-partitions, and an action type of the modification for each row modified.