Generation and traversal of a hierarchical index structure for efficient data retrieval
Methods, systems, apparatuses, and computer program products are described herein for the generation and traversal of a hierarchical index structure. The structure indexes search keys from data ingested from different data sources and enables efficient retrieval of the keys. As data is ingested, index nodes are generated at the lowest level of the structure. The nodes are analyzed to determine whether such nodes comprise duplicate keys. Responsive to doing so, a new index node is generated located at a higher level of the structure. This process results in a DAG comprising orphan nodes including different search keys. When processing a query for search keys, the orphan index nodes are initially analyzed for the keys. Upon finding a search key, its child nodes are recursively searched until location information specifying the location of ingested data in which the search key is located is found.
1. A method, comprising:
receiving a search query comprising a search key; and
traversing a hierarchical index structure comprising a plurality of index nodes for the search key, said traversing comprising:
determining that a first orphan index node of the plurality of index nodes or a first child index node of the first orphan index node does not comprise the search key;
responsive to determining that the first orphan index node or the first child index node does not comprise the search key, analyzing a second orphan index node of the plurality of index nodes located at the highest level or at a lower level of the hierarchical index structure for the search key; and
responsive to determining that second orphan index node or a second child index node of the second orphan index node comprises the search key, retrieving the data record comprising the search key from a data block referenced by the second orphan index node or the second child index node and returning the data record in a response to the search query.
2. The method of claim 1 , further comprising:
maintaining a progress log that stores a first indication of each of the plurality of index nodes that have been generated for the hierarchical index structure.
3. The method of claim 2 , wherein the progress log further comprises a second indication of data blocks that have been stored in a file system but for which an index node has not yet been generated for the hierarchical index structure.
4. The method of claim 3 , further comprising:
determining that at least one data block of the data blocks comprises the search key;
retrieving the search key from the at least one data block; and
returning the search key retrieved from the at least one data block in the response to the search query.
5. The method of claim 4 , wherein said determining comprises:
performing a linear scan operation on the data blocks.
6. The method of claim 1 , wherein the data block is referenced by the first orphan index node or the first child index node via location information maintained by the first orphan index node or the first child index node.
7. The method of claim 1 , wherein the location information comprises a uniform resource identifier identifying at least one of a path to a file or an offset thereof at which the data block is stored.
8. The method of claim 1 , wherein the first orphan index node and the second orphan index node are parentless.
9. A system, comprising:
at least one processor circuit; and
at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising:
a query processor configured to:
receive a search query comprising a search key; and
traverse a hierarchical index structure comprising a plurality of index nodes for the search key by:
analyzing a first orphan index node of the plurality of index nodes located at the highest level of the hierarchical index structure to determine whether the first orphan index node or a first child index node of the first orphan index node comprises the search key;
responsive to determining that first orphan index node or the first child index node comprises the search key, retrieving a data record comprising the search key from a data block referenced by the first orphan index node or the first child index node and returning the data record in a response to the search query;
responsive to determining that the first orphan index node or the first child index node does not comprise the search key, analyzing a second orphan index node of the plurality of index nodes located at the highest level or at a lower level of the hierarchical index structure for the search key; and
responsive to determining that second orphan index node or the second child index node comprises the search key, retrieving the data record comprising the search key from a data block referenced by the second orphan index node or the second child index node and returning the data record in a response to the search query.
10. The system of claim 9 , wherein a progress log is maintained that stores a first indication of each of the plurality of index nodes that have been generated for the hierarchical index structure.
11. The system of claim 10 , wherein the progress log further comprises a second indication of data blocks that have been stored in a file system but for which an index node has not yet been generated for the hierarchical index structure.
12. The system of claim 11 , wherein the query processor is further configured to:
determine that at least one data block of the data blocks comprises the search key;
retrieve the search key from the at least one data block; and
return the search key retrieved from the at least one data block in the response to the search query.
13. The system of claim 12 , wherein the query processor determines that at least one data block of the data blocks comprises the search key by:
performing a linear scan operation on the data blocks.
14. The system of claim 9 , wherein the data block is referenced by the first orphan index node or the first child index node via location information maintained by the first orphan index node or the first child index node.
15. A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor, perform a method comprising:
receiving a search query comprising a search key; and
traversing a hierarchical index structure comprising a plurality of index nodes for the search key, said traversing comprising:
determining that a first orphan index node of the plurality of index nodes or a first child index node of the first orphan index node does not comprise the search key;
responsive to determining that the first orphan index node or the first child index node does not comprise the search key, analyzing a second orphan index node of the plurality of index nodes located at the highest level or at a lower level of the hierarchical index structure for the search key; and
responsive to determining that second orphan index node or a second child index node of the second orphan index node comprises the search key, retrieving the data record comprising the search key from a data block referenced by the second orphan index node or the second child index node and returning the data record in a response to the search query.
16. The computer-readable storage medium of claim 15 , the method further comprising:
maintaining a progress log that stores a first indication of each of the plurality of index nodes that have been generated for the hierarchical index structure.
17. The computer-readable storage medium of claim 16 , wherein the progress log further comprises a second indication of data blocks that have been stored in a file system but for which an index node has not yet been generated for the hierarchical index structure.
18. The computer-readable storage medium of claim 17 , the method further comprising:
determining that at least one data block of the data blocks comprises the search key;
retrieving the search key from the at least one data block; and
returning the search key retrieved from the at least one data block in the response to the search query.
19. The computer-readable storage medium of claim 18 , wherein said determining comprises:
performing a linear scan operation on the data blocks.
20. The computer-readable storage medium of claim 15 , wherein the data block is referenced by the first orphan index node or the first child index node via location information maintained by the first orphan index node or the first child index node.