Method and system for searching a key-value storage
The present teaching relates to a method, system and programming for searching a data storage. A key is extracted from a request and a metadata object associated with the key is identified. Further, a determination is made as to whether the metadata object is associated with a data structure stored in a first portion of the data storage. In response to a successful determination, the data structure is searched to retrieve a value associated with the key from the first portion. In response to an unsuccessful determination, a cache is searched to retrieve the value associated with the key, and in response to the key being absent in the cache, a file associated with the metadata object is searched to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.
1. A method, implemented on a machine having at least one processor, storage, and a communication platform capable of connecting to a network for searching a data storage, the method comprising:
extracting a key from a request;
identifying a metadata object associated with the key;
determining whether the metadata object is associated with a data structure stored in a first portion of the data storage;
searching, in response to a successful determination, the data structure to retrieve a value associated with the key from the first portion;
searching, in response to an unsuccessful determination, a cache to retrieve the value associated with the key; and
searching, in response to the key being absent in the cache, a file associated with the metadata object to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.
2. The method of claim 1 , wherein the cache is stored in the first portion of the data storage, and wherein the second portion of the data storage is separate from the first portion of the data storage.
3. The method of claim 1 , wherein the file includes a data log and a data table, and wherein the step of searching the file further comprises:
searching the log to retrieve the value associated with the key; and
searching, in response to the key being absent in the log, the table to retrieve the value associated with the key.
4. The method of claim 3 , wherein the step of searching the log to retrieve the value further comprises:
extracting a Bloom filter associated with the metadata object;
identifying a partition of a plurality of partitions of the Bloom filter based on the key; and
determining, based on the identified partition, a portion of the log that is to be searched.
5. The method of claim 4 , wherein the data structure associated with the metadata object is a linked list including a plurality of keys stored in a sorted order, and wherein the data structure is searched via a binary search operation.
6. The method of claim 1 , wherein the metadata object includes information associated with a contiguous range of keys, and wherein the step of identifying the metadata object further comprises determining, whether the metadata object is associated with the key based on information included in an index file.
7. The method of claim 6 , wherein the index file is stored in the first portion of the data storage and includes a mapping of keys to a plurality of metadata objects.
8. A machine readable and non-transitory medium having information recorded thereon for searching a data storage, wherein the information, when read by the machine, causes the machine to perform:
extracting a key from a request;
identifying a metadata object associated with the key;
determining whether the metadata object is associated with a data structure stored in a first portion of the data storage;
searching, in response to a successful determination, the data structure to retrieve a value associated with the key from the first portion;
searching, in response to an unsuccessful determination, a cache to retrieve the value associated with the key; and
searching, in response to the key being absent in the cache, a file associated with the metadata object to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.
9. The machine readable and non-transitory medium of claim 8 , wherein the cache is stored in the first portion of the data storage, and wherein the second portion of the data storage is separate from the first portion of the data storage.
10. The machine readable and non-transitory medium of claim 8 , wherein the file includes a data log and a data table, and wherein the step of searching the file further comprises:
searching the log to retrieve the value associated with the key; and
searching, in response to the key being absent in the log, the table to retrieve the value associated with the key.
11. The machine readable and non-transitory medium of claim 10 , wherein the step of searching the log to retrieve the value further comprises:
extracting a Bloom filter associated with the metadata object;
identifying a partition of a plurality of partitions of the Bloom filter based on the key; and
determining, based on the identified partition, a portion of the log that is to be searched.
12. The machine readable and non-transitory medium of claim 11 , wherein the data structure associated with the metadata object is a linked list including a plurality of keys stored in a sorted order, and wherein the data structure is searched via a binary search operation.
13. The machine readable and non-transitory medium of claim 8 , wherein the metadata object includes information associated with a contiguous range of keys, and wherein the step of identifying the metadata object further comprises determining, whether the metadata object is associated with the key based on information included in an index file.
14. The machine readable and non-transitory medium of claim 13 , wherein the index file is stored in the first portion of the data storage and includes a mapping of keys to a plurality of metadata objects.
15. A system for searching a data storage, the system comprising a processor and a memory including instructions executed by the processor to perform operations comprising:
extracting a key from a request;
identifying a metadata object associated with the key;
determining whether the metadata object is associated with a data structure stored in a first portion of the data storage;
searching, in response to a successful determination, the data structure to retrieve a value associated with the key from the first portion,
searching, in response to an unsuccessful determination, a cache to retrieve the value associated with the key, and
searching, in response to the key being absent in the cache, a file associated with the metadata object to retrieve the value associated with the key, wherein the file is stored in a second portion of the data storage.
16. The system of claim 15 , wherein the cache is stored in the first portion of the data storage, and wherein the second portion of the data storage is separate from the first portion of the data storage.
17. The system of claim 15 , wherein the file includes a data log and a data table, and wherein the instructions are executed by the processor to further perform operations comprising:
searching the log to retrieve the value associated with the key; and
searching, in response to the key being absent in the log, the table to retrieve the value associated with the key.
18. The system of claim 17 , wherein the instructions are executed by the processor to further perform operations comprising:
extracting a Bloom filter associated with the metadata object;
identifying a partition of a plurality of partitions of the Bloom filter based on the key; and
determining, based on the identified partition, a portion of the log that is to be searched.
19. The system of claim 18 , wherein the data structure associated with the metadata object is a linked list including a plurality of keys stored in a sorted order, and wherein the data structure is searched via a binary search operation.
20. The system of claim 15 , wherein the metadata object includes information associated with a contiguous range of keys, and wherein the step of identifying the metadata object further comprises determining, whether the metadata object is associated with the key based on information included in an index file.