Hash engine for conducting point queries
Systems and methods are provided for improved point querying of a database. The index values are separated from data and retained in cache memory to allow access without requiring a disk input/output (I/O) operation and thereby having less latency resulting from such disk I/O operations. The index values can be compressed using an algorithm such as Crit-Bit-Trie to allow storage of the index values in limited cache memory space. The index values can be selected for storage according to a least recently used approach when cache memory is insufficient to store all index values to maintain a hit rate for the cached portion and reduce the disk I/O operations.
1. A database system, comprising:
one or more memories storing a database;
a cache memory; and
one or more processors, wherein the one or more processors together are configured to:
obtain index information from the database, wherein the index information is separate from a data component of the database;
generate a compact index based on the index information, wherein the compact index is smaller in size than the index information, the compact index generated by at least one of compression of the index information and omission of a portion of the index information from the compact index;
direct the storage of the compact index in the cache memory;
receive a point query;
identify data responsive to the point query by referencing the compact index stored in the cache memory; and
retrieve the data responsive to the point query from the database.
2. The database system of claim 1 , wherein the index information includes a first-level index mapping keys to blob file numbers and a second-level index mapping keys to block offsets within a blob file.
3. The database system of claim 1 , wherein the one or more processors are configured to perform a compression of the index information using a compression algorithm selected from the group consisting of Zstandard, LZ4, Snappy, Bzip2, LZMA, Gzip, Blosc, and Crit-Bit-Trie compression algorithms.
4. The database system of claim 3 , wherein the compression algorithm is Crit-Bit-Trie.
5. The database system of claim 1 , wherein the one or more processors are configured to select the portion omitted from the index information based on a least recently used strategy.
6. The database system of claim 1 , wherein the portion of the index information omitted from the compact index is stored in the one or more memories, and the one or more processors are configured to obtain the omitted index information from the one or more memories when the index information stored in the cache memory is not responsive to the point query.
7. The database system of claim 1 , wherein the compact index stored in the cache memory includes all index metadata for the index information.
8. The database system of claim 1 , wherein the one or more processors are configured to perform cache warmup when the database is started up, the cache warmup including obtaining initial index data and storing the initial index data in the cache memory.
9. The database system of claim 1 , wherein the one or more processors are configured to identify the data responsive to the point query and to retrieve the data responsive to the point query from the one or more memories storing the database using one single input/output operation.
10. A method for point querying of a database, comprising:
obtaining index information from the database, the index information being separate from a data component of the database;
generating, based on the index information, a compact index using a processor, wherein generating the compact index includes at least one of compressing the index information and omitting a portion of the index information from the compact index;
storing the index information in a cache memory;
receiving a point query;
identifying data responsive to the point query by referencing the index information stored in the cache memory; and
retrieving the responsive data from the database.
11. The method of claim 10 , wherein the index information includes a first-level index mapping keys to blob file numbers and a second-level index mapping keys to block offsets within a blob file.
12. The method of claim 10 , wherein generating the compact index includes compressing the index information using a compression algorithm selected from the group consisting of Zstandard, LZ4, Snappy, Bzip2, LZMA, Gzip, Blosc, and Crit-Bit-Trie compression algorithms.
13. The method of claim 12 , wherein the compression algorithm is Crit-Bit-Trie.
14. The method of claim 10 , wherein generating the compact index includes omitting the portion of the index information from the compact index, wherein said portion of the index information is determined based on a least recently used strategy.
15. The method of claim 14 , wherein when the index information stored in the cache memory is not responsive to the point query, the method further includes accessing the database to obtain the omitted portion of the index information.
16. The method of claim 10 , further comprising performing cache warmup when the database is started up, wherein the cache warmup includes obtaining initial index data and storing the initial index data in the cache memory.
17. The method of claim 10 , wherein the identifying of data responsive to the point query and the retrieving of the responsive data from the database is performed using one single input/output operation.
18. The method of claim 10 , wherein the compact index stored in the cache memory includes all index metadata for the index information.
19. A hash engine system for a database, comprising one or more processors configured to:
receive index information, wherein the index information is separate from a data component of the database;
generate a compact index based on the index information, wherein generating the compact index includes at least one of compression of the index information and omission of a portion of the index information from the compact index; and
direct storage of the compact index in a cache memory.
20. The hash engine system of claim 19 , wherein the one or more processors are further configured to:
receive a point query;
reference the compact index to determine data responsive to the point query; and
based on the compact index, obtain the data responsive to the point query from the database.