SYSTEM AND METHOD FOR SUPPORTING VERY LARGE DATA SETS IN DATABASES
A method and system of querying a partitioned data set organized in a static hierarchy. The method includes receiving a query for data in the partitioned data set at a first node, applying the query to a virtual data set of the first node to determine at least one child node having a child virtual data set with data for the query, generating at least one sub query for the at least one child node, where the at least one sub query is based on the query, and sending the at least one sub query to the at least one child to process.
1 . A method of querying a partitioned data set organized in a static hierarchy, the method comprising:
receiving a query for data in the partitioned data set at a first node;
applying the query to a virtual data set of the first node to determine at least one child node having a child virtual data set with data for the query;
generating at least one sub query for the at least one child node, where the at least one sub query is based on the query; and
sending the at least one sub query to the at least one child to process.
2 . The method of claim 1 , wherein the virtual data set of the first node includes a register of data in the partitioned data set that is managed by a plurality of child nodes of the first node.
3 . The method of claim 1 , further comprising;
receiving data returned from the at least one sub query; and
combining the data returned from the at least one sub query with data returned from other sub queries.
4 . The method of claim 3 , further comprising:
applying the query to the combined data; and
returning the result of the query to a requestor.
5 . A method of storing a partitioned data set organized in a static hierarchy, the method comprising:
receiving a large data set to be stored in data management system;
partitioning the large data set into a set of fixed sized partitions;
registering data in each of the set of fixed size partitions as a set of virtual data sets;
storing each partition of the set of fixed size partitions in the data management system; and
storing the set of virtual data sets in the data management system.
6 . The method of claim 5 , wherein the set of virtual data sets have a fixed hierarchy.
7 . The method of claim 6 , wherein the fixed hierarchy is a tree structure having a root node with a virtual data set with registered data for each child virtual data set.
8 . The method of claim 6 , wherein leaves of the tree structure are the fixed sized partitions.
9 . A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor, will cause the processor to perform operations of a method of querying a partitioned data set organized in a static hierarchy, the operations comprising:
receiving a query for data in the partitioned data set at a first node;
applying the query to a virtual data set of the first node to determine at least one child node having a child virtual data set with data for the query;
generating at least one sub query for the at least one child node, where the at least one sub query is based on the query; and
sending the at least one sub query to the at least one child to process.
10 . The non-transitory machine-readable storage medium of claim 9 , wherein the virtual data set of the first node includes a register of data in the partitioned data set that is managed by a plurality of child nodes of the first node.
11 . The non-transitory machine-readable storage medium of claim 9 , that provides further instructions that, if executed by a processor, will cause the processor to perform operations of a method of querying a partitioned data set organized in a static hierarchy, the operations comprising:
receiving data returned from the at least one sub query; and
combining the data returned from the at least one sub query with data returned from other sub queries.
12 . The non-transitory machine-readable storage medium of claim 11 that provides instructions that, if executed by a processor, will cause the processor to perform operations of a method of querying a partitioned data set organized in a static hierarchy, the operations comprising:
applying the query to the combined data; and
returning the result of the query to a requestor.
13 . The non-transitory machine-readable storage medium of claim 9 , that provides further instructions that, if executed by a processor, will cause the processor to perform operations of a method of querying a partitioned data set organized in a static hierarchy, the operations comprising:
receiving a large data set to be stored in data management system;
partitioning the large data set into a set of fixed sized partitions;
registering data in each of the set of fixed size partitions as a set of virtual data sets;
storing each partition of the set of fixed size partitions in the data management system; and
storing the set of virtual data sets in the data management system.
14 . The non-transitory machine-readable storage medium of claim 13 , wherein the set of virtual data sets have a fixed hierarchy.
15 . The non-transitory machine-readable storage medium of claim 14 , wherein the fixed hierarchy is a tree structure having a root node with a virtual data set with registered data for each child virtual data set.
16 . The non-transitory machine-readable storage medium of claim 14 , wherein leaves of the tree structure are the fixed sized partitions.
17 . A computing device to execute a method of querying a partitioned data set organized in a static hierarchy, the computing device comprising:
a non-transitory machine-readable medium having stored therein a query manager; and
a processor coupled to the non-transitory machine-readable medium, the processor to execute the query manager, the query manager to receive a query for data in the partitioned data set at a first node, apply the query to a virtual data set of the first node to determine at least one child node having a child virtual data set with data for the query, generate at least one sub query for the at least one child node, where the at least one sub query is based on the query, and send the at least one sub query to the at least one child to process.
18 . The computing device of claim 17 , wherein the virtual data set of the first node includes a register of data in the partitioned data set that is managed by a plurality of child nodes of the first node.
19 . The computing device of claim 17 , wherein the query manger is further to receive data returned from the at least one sub query, and combine the data returned from the at least one sub query with data returned from other sub queries.
20 . The computing device of claim 19 , wherein the query manager is further to apply the query to the combined data, and return the result of the query to a requestor.
21 . A computing device to implement a method of storing a partitioned data set organized in a static hierarchy, the computing device comprising:
a non-transitory machine-readable storage medium having stored therein a partition manager; and
a processor coupled to the non-transitory machine-readable storage medium, the processor to execute the partition manager, the partition manager to receive a large data set to be stored in data management system, partition the large data set into a set of fixed sized partitions, register data in each of the set of fixed size partitions as a set of virtual data sets, store each partition of the set of fixed size partitions in the data management system, and store the set of virtual data sets in the data management system.
22 . The computing device of claim 21 , wherein the set of virtual data sets have a fixed hierarchy.
23 . The computing device of claim 22 , wherein the fixed hierarchy is a tree structure having a root node with a virtual data set with registered data for each child virtual data set.
24 . The computing device of claim 22 , wherein leaves of the tree structure are the fixed sized partitions.