Indexing cloud database logs in a distributed database store
In an example embodiment, a transaction log index is used to index log records. The transaction log index comprises an in-memory transaction log index and an on-disk transaction log index. The log records are temporarily indexed in the in-memory transaction log index until the storage server can ensure that the corresponding log records are stored without any missing log records in a sequence, as well as possibly based on other correctness and performance considerations.
1 . A system comprising:
at least one processor; and
memory comprising instructions that, when executed by the at least one processor, cause the system to perform operations comprising:
receiving a request to write a transaction log pertaining to a plurality of database transactions, the transaction log having a plurality of log records, each log record having an identifier;
asynchronously storing the transaction log in a disk-based log store;
indexing the plurality of log records in an in-memory transaction log index;
in response to a determination that the plurality of log records are not missing any log records in an ordered sequence of log records, updating an on-disk transaction log index in the disk-based log store by storing log record identifiers in the on-disk transaction log index in order; and
deleting the plurality of log records in the ordered sequence from the in-memory transaction log index, to allow in-memory index capacity to be reclaimed after an ordered on-disk update.
2 . The system of claim 1 , wherein the updating of the on-disk transaction log index is further performed in response to at least one of one or more additional correctness and one or more performance considerations.
3 . The system of claim 1 , wherein the updating further comprises persisting all log record identifiers equal to or below a certain value to the on-disk transaction log index while maintaining all log record identifiers above the certain value in the in-memory transaction log index, and wherein the deleting the log record identifiers comprises deleting log record identifiers equal to or below the certain value from the in-memory transaction log index.
4 . The system of claim 1 , wherein the on-disk transaction log index is two levels, a root node and a plurality of leaf nodes, wherein the root node contains root node entries, each root node entry pointing to a leaf node containing log record identifiers within a specified range.
5 . The system of claim 4 , wherein each entry in the leaf nodes identifies a range of log records in the disk-based log store.
6 . The system of claim 4 , wherein existing root and leaf nodes are reused without allocating or freeing root or leaf nodes.
7 . The system of claim 6 , wherein no split or merge operation is performed on nodes of the on-disk transaction log index.
8 . The system of claim 6 , wherein no merge or compaction operation is performed across levels of the on-disk transaction log index and there is no amplification of input or output operations in the on-disk transaction log index.
9 . A machine-storage medium having instructions embodied thereon, the instructions executable by a processor of a machine to perform operations comprising:
receiving a request to write a transaction log pertaining to a plurality of database transactions, the transaction log having a plurality of log records, each log record having an identifier;
asynchronously storing the transaction log in a disk-based log store;
indexing the plurality of log records in an in-memory transaction log index;
in response to a determination that the plurality of log records are not missing any log records in an ordered sequence of log records, updating an on-disk transaction log index in the disk-based log store by storing log record identifiers in the on-disk transaction log index in order; and
deleting the plurality of log records in the ordered sequence from the in-memory transaction log index, to allow in-memory index capacity to be reclaimed after an ordered on-disk update.
10 . The machine-storage medium of claim 9 , wherein the updating of the on-disk transaction log index is further performed in response to at least one of one or more additional correctness and one or more performance considerations.
11 . The machine-storage medium of claim 9 , wherein the updating further comprises persisting all log record identifiers equal to or below a certain value to the on-disk transaction log index while maintaining all log record identifiers above the certain value in the in-memory transaction log index, and wherein the deleting the log record identifiers comprises deleting log record identifiers equal to or below the certain value from the in-memory transaction log index.
12 . The machine-storage medium of claim 9 , wherein the on-disk transaction log index is two levels, a root node and a plurality of leaf nodes, wherein the root node contains root node entries, each root node entry pointing to a leaf node containing log record identifiers within a specified range.
13 . The machine-storage medium of claim 12 , wherein each entry in the leaf nodes identifies a range of log records in the disk-based log store.
14 . The machine-storage medium of claim 12 , wherein existing root and leaf nodes are reused without allocating or freeing root or leaf nodes.
15 . The machine-storage medium of claim 14 , wherein no split or merge operation is performed on nodes of the on-disk transaction log index.
16 . The machine-storage medium of claim 14 , wherein no merge or compaction operation is performed across levels of the on-disk transaction log index and there is no amplification of input or output operations in the on-disk transaction log index.
17 . A method comprising:
receiving a request to write a transaction log pertaining to a plurality of database transactions, the transaction log having a plurality of log records, each log record having an identifier;
asynchronously storing the transaction log in a disk-based log store;
indexing the plurality of log records in an in-memory transaction log index;
in response to a determination that the plurality of log records are not missing any log records in an ordered sequence of log records, updating an on-disk transaction log index in the disk-based log store by storing log record identifiers in the on-disk transaction log index in order; and
deleting the plurality of log records in the ordered sequence from the in-memory transaction log index, to allow in-memory index capacity to be reclaimed after an ordered on-disk update.
18 . The method of claim 17 , wherein the updating of the on-disk transaction log index is further performed in response to at least one of one or more additional correctness and one or more performance considerations.
19 . The method of claim 17 , wherein the updating further comprises persisting all log record identifiers equal to or below a certain value to the on-disk transaction log index while maintaining all log record identifiers above the certain value in the in-memory transaction log index, and wherein the deleting the log record identifiers comprises deleting log record identifiers equal to or below the certain value from the in-memory transaction log index.
20 . The method of claim 17 , wherein the on-disk transaction log index is two levels, a root node and a plurality of leaf nodes, wherein the root node contains root node entries, each root node entry pointing to a leaf node containing log record identifiers within a specified range.