Variable cardinality index and data retrieval
Systems and methods for a variable cardinality index and data retrieval. A multi-level hash table increases cardinality from a top index and comprises multiple levels respectively pointing to at least one other level of hash tables. Each hash table comprises a unique file identification for a record in the index. A search engine module receives a token and cardinality limit, and performs a search through the multi-level hash table index for records associated with the token. The search is limited by the cardinality limit, and comprises a hash table lookup and a B-Tree progression. In response to a hit in the token search, the search engine performs a subsequent B-Tree progression from the hash table in which the hit occurred. The search engine returns the record, if encountered in the subsequent B-Tree progression. The unique file identification is returned if missing data is encountered, or the cardinality limit is reached.
1. A variable cardinality indexing and retrieval system comprising:
a computing device comprising a processor and at least one memory in communication with the processor, the at least one memory comprising instructions executable by the processor to:
execute a multi-level hash table index comprising multiple levels of hash tables respectively pointing to at least one other level of hash tables, wherein the multi-level hash table index increases cardinality from a top index, at least one portion of the hash tables in the multi-level hash table index points to a subset comprising a multitude of records, and each hash table includes at least one unique file identification for a record in the multitude of records; and
execute a search engine receiving a token and a cardinality limit, and performing a search through the multi-level hash table index for one or more records associated with the token, wherein the search comprises a hash table lookup and a B-Tree progression, the search engine performs a token search on each hash table through the B-tree progression, and the search is limited according to the cardinality limit.
2. The system of claim 1 , wherein, in response to a hit in the token search, the search engine performs a subsequent B-Tree progression from the hash table in which the hit occurred, wherein the subsequent B-Tree progression searches for a record associated with the hit, and the subsequent B-Tree progression remains limited according to the cardinality limit.
3. The system of claim 2 , wherein, in response to the search engine encountering the record in the subsequent B-Tree progression, the search engine returns the respective record.
4. The system of claim 2 , wherein, in response to the search engine encountering missing data in place of the record, the search engine returns the unique file identification for the record.
5. The system of claim 2 , wherein, in response to the search engine reaching the cardinality limit before encountering the record, the search engine returns the unique file identification for the record.
6. The system of claim 1 , wherein the token is one or more of: an IP address, an email address, a domain identifier, a character string, and an index file identifier.
7. A computer-implemented method, comprising: receiving, at a search engine, a token and a cardinality limit; and performing a search through a multi-level hash table index for one or more records associated with the token, wherein the search comprises a hash table lookup and a B-Tree progression, and the search is limited according to the cardinality limit, wherein the multi-level hash table index increases cardinality from a top index and comprises multiple levels of hash tables respectively pointing to at least one other level of hash tables, wherein at least one portion of the hash tables in the multi-level hash table index points to a subset comprising a multitude of records, the search engine performs a token search on each hash table through the B-tree progression and each hash table includes at least one unique file identification for a record in the multitude of records.
8. The method of claim 7 , wherein, in response to a hit in the token search, the search engine performs a subsequent B-Tree progression from the hash table in which the hit occurred, wherein the subsequent B-Tree progression searches for a record associated with the hit, and the subsequent B-Tree progression remains limited according to the cardinality limit.
9. The method of claim 8 , wherein, in response to the search engine encountering the record in the subsequent B-Tree progression, the search engine returns the respective record.
10. The method of claim 8 , wherein, in response to the search engine encountering missing data in place of the record, the search engine returns the unique file identification for the record.
11. The method of claim 8 , wherein, in response to the search engine reaching the cardinality limit before encountering the record, the search engine returns the unique file identification for the record.
12. The method of claim 7 , wherein the token is one or more of: an IP address, an email address, a domain identifier, a character string, and an index file identifier.
13. A non-transitory computer readable storage medium comprising instructions that, when executed on a computing system, cause the computing system to at least: receive, at a search engine, a token and a cardinality limit; and perform a search through a multi-level hash table index for one or more records associated with the token, wherein the search comprises a hash table lookup and a B-Tree progression, and the search is limited according to the cardinality limit, wherein the multi-level hash table index increases cardinality from a top index and comprises multiple levels of hash tables respectively pointing to at least one other level of hash tables, wherein at least one portion of the hash tables in the multi-level hash table index points to a subset comprising a multitude of records, the search engine performs a token search on each hash table through the B-tree progression, and each hash table includes at least one unique file identification for a record in the multitude of records.
14. The non-transitory computer readable storage medium of claim 13 , wherein, in response to a hit in the token search, the search engine performs a subsequent B-Tree progression from the hash table in which the hit occurred, wherein the subsequent B-Tree progression searches for a record associated with the hit, and the subsequent B-Tree progression remains limited according to the cardinality limit.
15. The non-transitory computer readable storage medium of claim 14 , wherein, in response to the search engine encountering the record in the subsequent B-Tree progression, the search engine returns the respective record.
16. The non-transitory computer readable storage medium of claim 14 , wherein, in response to the search engine encountering missing data in place of the record, the search engine returns the unique file identification for the record.
17. The non-transitory computer readable storage medium of claim 14 , wherein, in response to the search engine reaching the cardinality limit before encountering the record, the search engine returns the unique file identification for the record.