Compacting data file histories
Methods, systems, and apparatus for obtaining one or more metadata files, determining, by one or more computers and in accordance with a minor compaction setting, to perform a minor compaction of the one or more metadata files, creating one or more intermediate metadata files that each include at least compacted contacts of one or more of the metadata files, according to the determination to perform minor compaction of the one or more metadata files, determining, in accordance with a major compaction setting, to perform a major compaction of one or more of the intermediate metadata files, and creating one or more snapshot metadata files that each include at least compacted contents of one or more of the intermediate metadata files, according to the determination to perform the major compaction of one or more of the intermediate metadata files.
1. A system comprising:
a plurality of computers; and
a non-transitory storage device storing instructions operable to cause the one or more computers to perform operations comprising:
receiving a data modification request, the data modification request comprising a database operation statement for changing one or more data records in a database table, wherein the records are stored on multiple servers, each server including a computer program executing on one or more of the plurality of computers;
recording, by at least one of the servers, one or more entries in multiple operation log files, each entry in a log file associated with a respective server representing a history of changes to a data record stored on that server, the operation log files being stored on a storage device associated with that server;
determining, by a particular server among the at least one server, that a minor compaction condition has been satisfied;
in response to determining that the minor compaction condition has been satisfied, compacting the operation log files associated with the particular server, including merging at least two of the operation log files to form two or more merged operation log files, wherein compacting the operation log files associated with the particular server to form the two or more merged operation log files comprises discarding entries in the operation log files merged to form the merged operation log file, wherein a number of discarded entries is determined based on a minor compaction setting;
determining, by the particular server, that a major compaction condition has been satisfied; and
in response to determining that the major compaction condition has been satisfied, compacting the two or more merged operation log files, including merging at least two of the merged operation log files to form one persistence file, the persistence file including entries that represent a latest value for one or more of the data records in the database table.
2. The system of claim 1 , wherein each server is a Java virtual machine (JVM).
3. The system of claim 1 , wherein the database operation statement is a structured query language (SQL) statement for inserting, deleting, or updating data records in the database table, or for adding, deleting, truncating, or modifying columns of the database table.
4. The system of claim 1 , wherein the minor compaction condition specifies at least one of:
a file size condition specifying that the server should compact the at least two operation log files upon determining that a size of the at least two operation log files or that a size of each of the at least two operation log files satisfies a size condition,
a file number condition specifying that the server should compact the at least two operation log files when a number of operation log files reaches a threshold number, or
a time condition specifying that the server should compact the at least two operation log files upon determining that a threshold time since a last compaction has passed.
5. The system of claim 1 , wherein the minor compaction setting specifies that the operation log files are compacted in a minor compaction mode, wherein in the minor compaction mode:
the server merges eligible operation log files based on a minimum file size setting, a maximum file size setting, or a total file size setting, wherein a number of entries to be discarded during merging is determined based at least in part on the minimum file size setting, the maximum file size setting, or the total file size setting, and
the server deletes the eligible operation log after the merging after a period of time specified by a setting.
6. The system of claim 1 , wherein the major compaction setting specifies that the operation log files are compacted in a major compaction mode, wherein in the major compaction mode:
the server combines data record values in a logical bucket of the server into the persistence file, wherein a latest value of each data record is preserved in the persistence file, and wherein historical values of each data record are discarded.
7. The system of claim 1 , wherein the operation log files associated with the server include data identifying one or more rows or key-value pairs, wherein each row or key-value pair is associated with a data value in the database table.
8. A non-transitory storage device storing instructions operable to cause one or more computers to perform operations comprising:
receiving a data modification request, the data modification request comprising a database operation statement for changing one or more data records in a database table, wherein the records are stored on multiple servers, each server including a computer program executing on at least one of the one or more computers;
recording, by at least one of the servers, one or more entries in multiple operation log files, each entry in a log file associated with a respective server representing a history of changes to a data record stored on that server, the operation log files being stored on a storage device associated with that server;
determining, by a particular server among the at least one server, that a minor compaction condition has been satisfied;
in response to determining that the minor compaction condition has been satisfied, compacting the operation log files associated with the particular server, including merging at least two of the operation log files to form two or more merged operation log files, wherein compacting the operation log files associated with the particular server to form the two or more merged operation log files comprises discarding entries in the operation log files merged to form the merged operation log file, wherein a number of discarded entries is determined based on a minor compaction setting;
determining, by the particular server, that a major compaction condition has been satisfied; and
in response to determining that the major compaction condition has been satisfied, compacting the two or more merged operation log files, including merging at least two of the merged operation log files to form one persistence file, the persistence file including entries that represent a latest value for one or more of the data records in the database table.
9. The non-transitory storage device of claim 8 , wherein each server is a Java virtual machine (JVM).
10. The non-transitory storage device of claim 8 , wherein the database operation statement is a structured query language (SQL) statement for inserting, deleting, or updating data records in the database table, or for adding, deleting, truncating, or modifying columns of the database table.
11. The non-transitory storage device of claim 8 , wherein the minor compaction condition specifies at least one of:
a file size condition specifying that the server should compact the at least two operation log files upon determining that a size of the at least two operation log files or that a size of each of the at least two operation log files satisfies a size condition,
a file number condition specifying that the server should compact the at least two operation log files when a number of operation log files reaches a threshold number, or
a time condition specifying that the server should compact the at least two operation log files upon determining that a threshold time since a last compaction has passed.
12. The non-transitory storage device of claim 8 , wherein the minor compaction setting specifies that the operation log files are compacted in a minor compaction mode, wherein in the minor compaction mode:
the server merges eligible operation log files based on a minimum file size setting, a maximum file size setting, or a total file size setting, wherein a number of entries to be discarded during merging is determined based at least in part on the minimum file size setting, the maximum file size setting, or the total file size setting, and
the server deletes the eligible operation log after the merging after a period of time specified by a setting.
13. The non-transitory storage device of claim 8 , wherein the major compaction setting specifies that the operation log files are compacted in a major compaction mode, wherein in the major compaction mode:
the server combines data record values in a logical bucket of the server into the persistence file, wherein a latest value of each data record is preserved in the persistence file, and wherein historical values of each data record are discarded.
14. The non-transitory storage device of claim 8 , wherein the operation log files associated with the server include data identifying one or more rows or key-value pairs, wherein each row or key-value pair is associated with a data value in the database table.