System and a method for hierarchical data column storage and efficient query processing
An embodiment provides intermediate data derived in the form of column stores which are in turn based on hierarchical data stores. This intermediate data represents a reduced subset of data matched appropriately to a query (or modified query) such that the amount of data handled in a query processing task on large data is greatly reduced. An embodiment may appropriately choose column data stores and/or modify queries in order leverage parallelization techniques such as map-reduce in order to query large data. The result is the ability to query large data stores in parallel while reducing the amount of data that must be handled.
1. A method for data storage and searching, comprising:
utilizing at least one processor to execute computer code configured to perform the steps of:
creating column data based on a store of hierarchical data;
said creating comprising pre-processing the hierarchical data such that values of a column with respect to different rows are stored together;
accepting a data query with respect to the column data;
deriving hierarchy information from the store of hierarchical data;
generating key-value pairs based on each row of the column data;
the key-value pairs being stored, via map-reduce, in a reducer with similar key-value pairs to create intermediate data;
generating a modified query based on the data query and on an input data schema; and
executing the modified query using the intermediate data;
the hierarchical data comprising data organized as a plurality of tree-structures;
at least one of the tree-structures including one or more data attributes not relevant to the data query;
wherein the intermediate data do not include the at least one tree-structure having one or more data attributes not relevant to the data query.
2. The method of claim 1 , wherein said generating a modified query comprises changing the data query based on a data schema derived from the store of hierarchical data and a data schema of the intermediate data.
3. The method of claim 1 , wherein said executing comprises executing the modified query using a parallelized, distributed architecture.
4. The method of claim 1 , wherein the column data comprise data of the hierarchical data store organized into column-based data structures.
5. The method of claim 4 , wherein each of the column-based data structures comprises a plurality of record values and an attribute value associated with the plurality of record values.
6. The method of claim 5 , wherein each of the record values corresponds to hierarchical record data associated with the attribute value.
7. The method of claim 6 , wherein the intermediate data are created utilizing links between stores of the column data.
8. The method of claim 7 , wherein the links are derived from schemata of the store of hierarchical data.
9. A computer program product for data storage and searching, said computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
computer readable program code configured to create column data based on a store of hierarchical data, via pre-processing the hierarchical data such that values of a column with respect to different rows are stored together;
computer readable program code configured to accept a data query with respect to the column data;
computer readable program code configured to derive hierarchy information from the store of hierarchical data;
computer readable program code configured to generate key-value pairs based on each row of the column data;
the key-value pairs being stored, via map-reduce, in a reducer with similar key-value pairs to create intermediate data;
computer readable program code configured to generate a modified query based on the data query and on an input data schema; and
computer readable program code configured to execute the modified query using the intermediate data;
the hierarchical data comprising data organized as a plurality of tree-structures;
at least one of the tree-structures including one or more data attributes not relevant to the data query;
wherein the intermediate data do not include the at least one tree-structure having one or more data attributes not relevant to the data query.
10. The computer program product of claim 9 , wherein a modified query is generated by changing the data query based on a data schema derived from the store of hierarchical data and a data schema of the intermediate data.
11. The computer program product of claim 9 , wherein the modified query is executed by executing a data query using a parallelized, distributed architecture.
12. The computer program product of claim 9 , wherein the column data comprise data of the hierarchical data store organized into column-based data structures.
13. The computer program product of claim 12 , wherein each of the column-based data structures comprises a plurality of record values and an attribute value associated with the plurality of record values.
14. The computer program product of claim 13 , wherein each of the record values corresponds to hierarchical record data associated with the attribute value.
15. The computer program product of claim 14 , wherein the intermediate data are created utilizing links between stores of the column data.
16. The computer program product of claim 15 , wherein the links are derived from schema of the store of hierarchical data.
17. An apparatus for data storage and searching, said apparatus comprising:
at least one processor; and
a non-transitory computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising:
computer readable program code configured to create column data based on a store of hierarchical data, via pre-processing the hierarchical data such that values of a column with respect to different rows are stored together;
computer readable program code configured to accept a data query with respect to the column data;
computer readable program code configured to derive hierarchy information from the store of hierarchical data;
computer readable program code configured to generate key-value pairs based on each row of the column data;
the key-value pairs being stored, via map-reduce, in a reducer with similar key-value pairs to create intermediate data;
computer readable program code configured to generate a modified query based on the data query and on an input data schema; and
computer readable program code configured to execute the modified query using the intermediate data;
the hierarchical data comprising data organized as a plurality of tree-structures;
at least one of the tree-structures including one or more data attributes not relevant to the data query;
wherein the intermediate data do not include the at least one tree-structure having one or more data attributes not relevant to the data query.
18. A method comprising:
storing hierarchical data in a column storage;
generating key-value pairs based on each row of the column storage;
the key-value pairs being stored, via map-reduce, in a reducer with similar key-value pairs to create intermediate data;
compiling a query using a data schema to generate a modified query for the intermediate data; and
executing the modified query over the intermediate data;
said storing comprising:
storing each of a plurality of data attributes in separate columns; and
storing with each attribute at least one of: a node identification, a value, a parent identification and a record identification;
the hierarchical data comprising data organized as a plurality of tree-structures;
at least one of the tree-structures including one or more data attributes not relevant to the data query;
wherein the intermediate data do not include the at least one tree-structure having one or more data attributes not relevant to the data query.