Efficiently updating a secondary index associated with a log-structured merge-tree database
Systems and methods are described herein for efficiently updating a secondary index associated with a log-structured merge-tree (LSM) database. A Global approximate member query (AMQ) Filter is queried to determine whether a primary key, retrieved from a list of LSM database updates, already exists in the LSM database. If the primary key does not already exist in the LSM database then read-before-write and delete operations, typically performed with known approaches, do not need to be performed on the secondary index in order to update the secondary index, thereby avoiding significant additional computer processing and input/output operations.
1. A method for updating a secondary index associated with a log-structured merge-tree (LSM) database, the method comprising:
retrieving, by a computing system, a primary key from a list of updates made to the LSM database;
querying, by the computing system, a Global Approximate Member Query (AMQ) Filter to determine whether the retrieved primary key does or does not already exist in the LSM database;
updating the secondary index with the retrieved primary key without performing a read-before-write operation or a delete operation on the secondary index, when the result of the query is a determination that the retrieved primary key does not already exist in the LSM database; and,
updating the secondary index with the retrieved primary key by performing a read-before-write operation and a delete operation on the secondary index, when the result of the query is a determination that the retrieved primary key does already exist in the LSM database.
2. The method of claim 1 wherein the Global AMQ Filter is a probabilistic filter that supports deletes and can be resized without hashing.
3. The method of claim 1 wherein the Global AMQ Filter is a Quotient Filter.
4. The method of claim 1 wherein the Global AMQ Filter is a Cuckoo Filter.
5. A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for storing an index in working memory of a computing system that can concurrently be searched, the method comprising the steps of:
retrieving, by a computing system, a primary key from a list of updates made to the LSM database;
querying, by the computing system, a Global Approximate Member Query (AMQ) Filter to determine whether the retrieved primary key does or does not already exist in the LSM database;
updating the secondary index with the retrieved primary key without performing a read-before-write operation and a delete operation on the secondary index, when the result of the query is a determination that the retrieved primary key does not already exist in the LSM database; and,
updating the secondary index with the retrieved primary key by performing a read-before-write operation and a delete operation on the secondary index, when the result of the query is a determination that the retrieved primary key does already exist in the LSM database.
6. The non-transitory computer-readable storage medium having embodied thereon a program of claim 5 , wherein the Global AMQ Filter is a probabilistic filter that supports deletes and can be resized without hashing.
7. The non-transitory computer-readable storage medium having embodied thereon a program of claim 5 , wherein the Global AMQ Filter is a Quotient Filter.
8. The non-transitory computer-readable storage medium having embodied thereon a program of claim 5 , wherein the Global AMQ Filter is a Cuckoo Filter.
9. A computing system for updating a secondary index associated with a log-structured merge-tree (LSM) database, the computing system comprising:
a processor, a Global Approximate Member Query (AMQ) Filter, the LSM database, a list of updates made to the LSM database, and the secondary index;
wherein the processor is configured to:
retrieve a primary key from a list of updates made to a LSM database;
query the Global AMQ Filter to determine whether the retrieved primary key does or does not already exist in the LSM database;
update the secondary index with the retrieved primary key without performing a read-before-write operation or a delete operation on the secondary index, when the result of the processor query is a determination that the retrieved primary key does not already exist in the LSM database; and,
update the secondary index with the retrieved primary key by performing a read-before-write operation and a delete operation on the secondary index, when the result of the processor query is a determination that the retrieved primary key does already exist in the LSM database.
10. The computing system of claim 9 wherein the Global AMQ Filter is a probabilistic filter that supports deletes and can be resized without hashing.
11. The computing system of claim 9 wherein the Global AMQ Filter is a Quotient Filter.
12. The computing system of claim 9 wherein the Global AMQ Filter is a Cuckoo Filter.