Generating minor compactions to capture aggregated actions for commit ranges to data files

1. A method comprising:

receiving, from one or more client devices, requests to commit one or more transactions to a data table and write metadata for the one or more transactions to one or more log files, wherein the data table is stored in a cloud storage as a set of data files;

generating a checkpoint file aggregating the one or more transactions described in the one or more log files to compute a state of the data table at a first time;

receiving requests to commit a set of transactions to the data table and write metadata for the set of transactions to a set of log files, wherein the set of transactions are transactions that occur after the one or more transactions;

responsive to determining that a number of log files in the set of log files reaches a threshold commit number, generating a minor compaction file aggregating the set of transactions in the set of log files; and

generating a second checkpoint file aggregating the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file to compute a state of the data table at a second time.

2. The method of claim 1 , further comprising:

generating a set of first-level minor compaction files including the minor compaction file, each first-level minor compaction file aggregating a respective set of transactions; and

responsive to determining that a number of first-level minor compaction files reaches a threshold number, generating a second-level minor compaction file aggregating the set of transactions in the set of first-level minor compaction files.

3. The method of claim 1 , further comprising:

iteratively creating hierarchical minor compaction files comprising multiple levels of minor compaction files, each higher level minor compaction file aggregating information on a respective set of transactions obtained from a respective set of lower level minor compaction files.

4. The method of claim 3 , further comprising:

determining a number of minor compaction levels (k) based on a threshold number of files (n) that each minor compaction aggregates and a threshold number of files (m) to read from the cloud storage.

5. The method of claim 1 , wherein generating the second checkpoint file comprises:

aggregating the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file and an additional set of transactions in an additional set of log files; and

computing the state of the data table at the second time based on the aggregation.

6. The method of claim 1 , wherein the set of log files are in a form of JSON files.

7. The method of claim 1 , wherein each log file records one or more actions performed to the data table in a transaction corresponding to the log file.

8. A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed cause a processor system to:

receive, from one or more client devices, requests to commit one or more transactions to a data table and write metadata for the one or more transactions to one or more log files, wherein the data table is stored in a cloud storage as a set of data files;

generate a checkpoint file aggregating the one or more transactions described in the one or more log files to compute a state of the data table at a first time;

receive requests to commit a set of transactions to the data table and write metadata for the set of transactions to a set of log files;

responsive to determining that a number of log files in the set of log files reaches a threshold commit number, generate a minor compaction file aggregating the set of transactions in the set of log files; and

generate a second checkpoint file aggregating the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file to compute a state of the data table at a second time.

9. The non-transitory computer readable storage medium of claim 8 , wherein the instructions when executed further cause the processor system to:

generate a set of first-level minor compaction files including the minor compaction file, each first-level minor compaction file aggregating a respective set of transactions; and

responsive to determining that a number of first-level minor compaction files reaches a threshold number, generate a second-level minor compaction file aggregating the set of transactions in the set of first-level minor compaction files.

10. The non-transitory computer readable storage medium of claim 8 , wherein the instructions when executed further cause the processor system to:

iteratively creating hierarchical minor compaction files comprising multiple levels of minor compaction files, each higher level minor compaction file aggregating information on a respective set of transactions obtained from a respective set of lower level minor compaction files.

11. The non-transitory computer readable storage medium of claim 10 , wherein the instructions when executed further cause the processor system to:

determine a number of minor compaction levels based on a threshold number and a threshold number of files to read from the cloud storage.

12. The non-transitory computer readable storage medium of claim 8 , wherein the instruction to generate the second checkpoint file, when executed further cause the processor system to:

aggregate the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file and an additional set of transactions in an additional set of log files; and

compute the state of the data table at the second time based on the aggregation.

13. The non-transitory computer readable storage medium of claim 8 , wherein the set of log files are in a form of JSON files.

14. The non-transitory computer readable storage medium of claim 8 , wherein each log file records one or more actions performed to the data table in a transaction corresponding to the log file.

15. A system comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the system to:

receive, from one or more client devices, requests to commit one or more transactions to a data table and write metadata for the one or more transactions to one or more log files, wherein the data table is stored in a cloud storage as a set of data files;

generate a checkpoint file aggregating the one or more transactions described in the one or more log files to compute a state of the data table at a first time;

receive requests to commit a set of transactions to the data table and write metadata for the set of transactions to a set of log files;

responsive to determining that a number of log files in the set of log files reaches a threshold commit number, generate a minor compaction file aggregating the set of transactions in the set of log files; and

generate a second checkpoint file aggregating the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file to compute a state of the data table at a second time.

16. The system of claim 15 , wherein the instructions when executed by the one or more computer processors, further cause the system to:

generate a set of first-level minor compaction files including the minor compaction file, each first-level minor compaction file aggregating a respective set of transactions; and

responsive to determining that a number of first-level minor compaction files reaches a threshold number, generate a second-level minor compaction file aggregating the set of transactions in the set of first-level minor compaction files.

17. The system of claim 15 , wherein the instructions when executed by the one or more computer processors, further cause the system to:

iteratively create hierarchical minor compaction files comprising multiple levels of minor compaction files, each higher level minor compaction file aggregating information on a respective set of transactions obtained from a respective set of lower level minor compaction files.

18. The system of claim 17 , wherein the instructions when executed by the one or more computer processors, further cause the system to:

determine a number of minor compaction levels based on a threshold number and a threshold number of files to read from the cloud storage.

19. The system of claim 15 , wherein the instruction to generate the second checkpoint file, when executed by the one or more computer processors, further cause the system to:

aggregate the state of the data table at the first time with information on the set of transactions obtained from the minor compaction file and an additional set of transactions in an additional set of log files; and

compute the state of the data table at the second time based on the aggregation.

20. The system of claim 15 , wherein the set of log files are in a form of JSON files.