Scalable indexing architecture
Provided are methods and systems for improved data management, data analysis, and processing. A computing device may receive one or more global queries. Each of the global queries may be associated with one or more local components, such as individual nodes of a cloud network. The computing device may determine one or more indexlets associated with each local component. The computing device may then generate query results for the one or more global queries using a global query plan and the one or more indexlets.
1 . A method comprising:
receiving, by a computing device, updated data for a data model comprising a plurality of tables stored across a plurality of data sources, wherein the data model is associated with a hypercube that is generated based on the plurality of tables excluding the updated data;
determining, based on a global query associated with the updated data, a plurality of nodes associated with the plurality of data sources;
determining, for each node of the plurality of nodes, one or more indexlets, wherein each indexlet for the corresponding node is associated with a block of rows of a corresponding table, of the plurality of tables, associated with the updated data stored at the corresponding node, wherein each indexlet for the corresponding node comprises a row for each unique value of at least one attribute stored in the corresponding table and intersecting columns representing the block of rows of the corresponding table, and wherein each row for each unique value comprises a first character or a second character, the first character indicating the unique value is present in the row of the block of rows of the corresponding table that corresponds to the particular intersecting column of the indexlet, the second character indicating the unique value is not present in the row of the corresponding table that corresponds to the particular intersecting column of the indexlet;
determining, by each node of the plurality of nodes, based on the global query and each indexlet of the one or more indexlets, one or more data records stored in the block of rows of the corresponding table, wherein the one or more data records comprise each unique value of the at least one attribute; and
generating, based on the one or more data records associated with each node of the plurality of nodes, partial query results for the global query, wherein the partial query results are associated with the updated data and are generated without regenerating the hypercube.
2 . The method of claim 1 , wherein the plurality of data sources comprise at least one of: a plurality of databases, a plurality of computing devices, or a plurality of virtual machines, and wherein each node of the plurality of nodes comprises at least one of: a database of the plurality of databases, a computing device of the plurality of computing devices, or a virtual machine of the plurality of virtual machines.
3 . The method of claim 1 , wherein the global query comprises a plurality of portions, and wherein determining the one or more indexlets comprises:
determining, for each portion of the plurality of portions of the global query, at least one common subexpression.
4 . The method of claim 3 , further comprising:
determining, for each portion of the plurality of portions of the global query, based on the at least one common subexpression, a logical plan indicative of a traversal of the data model and the updated data for the data model.
5 . The method of claim 1 , wherein the global query comprises a plurality of portions, and wherein the method further comprises:
determining, for each portion of the plurality of portions of the global query, based on a logical plan indicative of a traversal of the data model and the updated data for the data model, a physical plan indicative of the plurality of nodes.
6 . The method of claim 5 , wherein the physical plan is further indicative of a location of each node of the plurality of node.
7 . The method of claim 1 , wherein the updated data comprises at least one of:
a modification to existing data within at least one table, of the plurality of tables, associated with the updated data; or
new data for at least one table, of the plurality of tables, associated with the updated data.
8 . A non-transitory computer-readable storage medium comprising computer-executable instructions that, when executed by a computing device, cause the computing device to:
receive updated data for a data model comprising a plurality of tables stored across a plurality of data sources, wherein the data model is associated with a hypercube that is generated based on the plurality of tables excluding the updated data;
determine, based on a global query associated with the updated data, a plurality of nodes associated with the plurality of data sources;
determine, for each node of the plurality of nodes, one or more indexlets, wherein each indexlet for the corresponding node is associated with a block of rows of a corresponding table, of the plurality of tables, associated with the updated data stored at the corresponding node, and wherein each indexlet for the corresponding node comprises a row for each unique value of at least one attribute stored in the corresponding table and intersecting columns representing the block of rows of the corresponding table, and wherein each row for each unique value comprises a first character or a second character, the first character indicating the unique value is present in the row of the block of rows of the corresponding table that corresponds to the particular intersecting column of the indexlet, the second character indicating the unique value is not present in the row of the corresponding table that corresponds to the particular intersecting column of the indexlet;
determine, by each node of the plurality of nodes, based on the global query and each indexlet of the one or more indexlets, one or more data records stored in the block of rows of the corresponding table, wherein the one or more data records comprise each unique value of the at least one attribute; and
generate, based on the one or more data records associated with each node of the plurality of nodes, partial query results for the global query, wherein the partial query results are associated with the updated data and are generated without regenerating the hypercube.
9 . The non-transitory computer-readable storage medium of claim 8 , wherein the plurality of data sources comprise at least one of: a plurality of databases, a plurality of computing devices, or a plurality of virtual machines, and wherein each node of the plurality of nodes comprises at least one of: a database of the plurality of databases, a computing device of the plurality of computing devices, or a virtual machine of the plurality of virtual machines.
10 . The non-transitory computer-readable storage medium of claim 8 , wherein the global query comprises a plurality of portions, and wherein the computer-executable instructions that cause the computing device to determine the one or more indexlets further cause the computing device to:
determine, for each portion of the plurality of portions of the global query, at least one common subexpression.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein the computer-executable instructions further cause the computing device to:
determine, for each portion of the plurality of portions of the global query, based on the at least one common subexpression, a logical plan indicative of a traversal of the data model and the updated data for the data model.
12 . The non-transitory computer-readable storage medium of claim 8 , wherein the global query comprises a plurality of portions, and wherein the computer-executable instructions further cause the computing device to:
determine, for each portion of the plurality of portions of the global query, based on a logical plan indicative of a traversal of the data model and the updated data for the data model, a physical plan indicative of the plurality of nodes.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the physical plan is further indicative of a location of each node of the plurality of nodes.
14 . The non-transitory computer-readable storage medium of claim 8 , wherein the updated data comprises at least one of:
a modification to existing data within at least one table, of the plurality of tables, associated with the updated data; or
new data for at least one table, of the plurality of tables, associated with the updated data.
15 . An apparatus comprising:
one or more processors; and
a memory comprising processor executable instructions that, when executed by the one or more processors, cause the apparatus to:
receive updated data for a data model comprising a plurality of tables stored across a plurality of data sources, wherein the data model is associated with a hypercube that is generated based on the plurality of tables excluding the updated data;
determine, based on a global query associated with the updated data, a plurality of nodes associated with the plurality of data sources;
determine, for each node of the plurality of nodes, one or more indexlets, wherein each indexlet for the corresponding node is associated with a block of rows of a corresponding table, of the plurality of tables, associated with the updated data stored at the corresponding node, wherein each indexlet for the corresponding node comprises a row for each unique value of at least one attribute stored in the corresponding table and intersecting columns representing the block of rows of the corresponding table, and wherein each row for each unique value comprises a first character or a second character, the first character indicating of whether the unique value is present in the row of the block of rows of the corresponding table that corresponds to the particular intersecting column of the indexlet, the second character indicating the unique value is not present in the row of the corresponding table that corresponds to the particular intersecting column of the indexlet;
determine, by each node of the plurality of nodes, based on the global query and each indexlet of the one or more indexlets, one or more data records stored in the block of rows of the corresponding table, wherein the one or more data records comprise each unique value of the at least one attribute; and
generate, based on the one or more data records associated with each node of the plurality of nodes, partial query results for the global query, wherein the partial query results are associated with the updated data and are generated without regenerating the hypercube.
16 . The apparatus of claim 15 , wherein the plurality of data sources comprise at least one of: a plurality of databases, a plurality of computing devices, or a plurality of virtual machines, and wherein each node of the plurality of nodes comprises at least one of: a database of the plurality of databases, a computing device of the plurality of computing devices, or a virtual machine of the plurality of virtual machines.
17 . The apparatus of claim 15 , wherein the global query comprises a plurality of portions, and wherein the processor executable instructions that cause the apparatus to determine the one or more indexlets further cause the apparatus to:
determine, for each portion of the plurality of portions of the global query, at least one common subexpression.
18 . The apparatus of claim 17 , wherein the processor executable instructions further cause the apparatus to:
determine, for each portion of the plurality of portions of the global query, based on the at least one common subexpression, a logical plan indicative of a traversal of the data model and the updated data for the data model; and
determine, for each portion of the plurality of portions of the global query, based on a logical plan indicative of a traversal of the data model and the updated data for the data model, a physical plan indicative of the plurality of nodes.
19 . The apparatus of claim 18 , wherein the physical plan is further indicative of a location of each node of the plurality of nodes.
20 . The apparatus of claim 15 , wherein the updated data comprises at least one of:
a modification to existing data within at least one table, of the plurality of tables, associated with the updated data; or
new data for at least one table, of the plurality of tables, associated with the updated data.