Systems and methods for intelligent capture and fast transformations of granulated data summaries in database engines
Embodiments may provide methods and systems for intelligent capture and fast transformation of granulated data summaries. An engine may be used to transform input data summaries into result sets representing query outcomes. The data summaries contain enough knowledge about the original data to accurately perform operations on the summaries without needing to access the original data. In an embodiment, the contents of data summaries are accessible via an SQL approximate engine which retrieves summaries stored on disk and utilizes them for its operations. Alternatively, the contents of data summaries are accessible via virtual tables which give users direct access to the summary contents and allow for the creation and implementation of algorithms to work with the data summaries independently from the SQL approximate engine.
1. A method for capture and transformation of granulated data summaries from a data set stored in a server implemented in a computer system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for the at least one chunk of data comprises:
dividing, at the computer system, the data stored in the server into a plurality of chunks;
generating, at the computer system, statistical information for at least some columns in some of the chunks of the data;
generating, at the computer system, statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating, at the computer system, at least one data summary based on the generated statistical information for the at least some columns and the generated statistical information about relationships between at least some pairs of columns; and
transmitting, at the computer system, the generated response to the query.
2. The method of claim 1 , wherein generating a response to the query comprises utilizing, at the computer system, a mechanism of tree-based belief propagation adapted to populate changes in the at least one data summary.
3. The method of claim 1 , wherein the generated response to the query is a standard SQL result.
4. A method for capture and transformation of granulated data summaries from a data set stored in a server implemented in a computer system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for at least one chunk of the data comprises:
receiving, at the computer system, data including a plurality of rows of the data at the computer system;
generating, at the computer system, at least a chunk of the data including a predetermined number of received rows of the data;
generating, at the computer system, statistical information for the generated chunks of the data;
generating, at the computer system, statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating, at the computer system, at least one data summary based on the generated statistical information for the generated chunks of data and on the generated statistical information about relationships between the at least some pairs of columns of the chunks of data; and
transmitting, at the computer system, the generated response to the query.
5. The method of claim 4 , wherein the statistical information for the generated chunks of data and the statistical information about relationships are updated using the at least one data summary that is generated as data is received by the computer system.
6. A method for capture and transformation of granulated data summaries from a data set stored in a server implemented in a computer system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein the at least one data summary of the at least one chunk of data takes the form of a relational data table with columns and rows, wherein the at least one data summary comprises at least one histogram reflecting intensities of occurrences of values of at least one column, and at least one co-occurrence ratio reflecting intensities of co-occurrences of values of a plurality of columns; and
transmitting, at the computer system, the generated response to the query.
7. The method of claim 6 , wherein the at least one histogram reflects intensities of occurrences within a quantized domain of values of the at least one column within the at least one data chunk, and wherein the at least one co-occurrence ratio reflects intensities of co-occurrences of quantized values of the plurality of columns.
8. The method of claim 7 , wherein the domain of values of the at least one column is quantized based on heuristic classification of the estimated expected accuracy of the generated response to the query subject to different choices of quantization settings.
9. The method of claim 7 , wherein the at least one co-occurrence ratio does not represent information about all possible combinations of values of the plurality of columns and the co-occurrence ratios related to unrepresented combinations of values are estimated based on information about the at least one co-occurrence ratio related to combinations of the quantized values.
10. The method of claim 7 , wherein the at least one co-occurrence ratio to be represented is selected from a set of all combinations of values of the plurality of columns based on heuristic classification of the estimated expected accuracy of the generated response to the query.
11. A system for capture and transformation of granulated data summaries from a data set stored in a server, the system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving a query for at least one chunk of data stored in the server;
generating a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for the at least one chunk of data comprises:
dividing, at the computer system, the data stored in the server into a plurality of chunks;
generating, at the computer system, statistical information for at least some columns in some of the chunks of the data;
generating, at the computer system, statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating at least one data summary based on the generated statistical information for the at least some columns and the generated statistical information about relationships between at least some pairs of columns; and
transmitting the generated response to the query.
12. The system of claim 11 , wherein generating a response to the query comprises utilizing a mechanism of tree-based belief propagation adapted to populate changes in the at least one data summary.
13. The system of claim 11 , wherein the generated response to the query is a standard SQL result.
14. A system for capture and transformation of granulated data summaries from a data set stored in a server, the system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving a query for at least one chunk of data stored in the server;
generating a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for at least one chunk of the data comprises:
receiving data including a plurality of rows of the data at the computer system;
generating at least a chunk of the data including a predetermined number of received rows of the data;
generating statistical information for the generated chunks of the data;
generating statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating at least one data summary based on the generated statistical information for the generated chunks of data and on the generated statistical information about relationships between the at least some pairs of columns of the chunks of data; and
transmitting the generated response to the query.
15. The system of claim 14 , wherein the statistical information for the generated chunks of data and the statistical information about relationships are updated using the at least one data summary that is generated as data is received by the computer system.
16. A system for capture and transformation of granulated data summaries from a data set stored in a server, the system comprising a processor, a memory adapted to store program instructions and data, and program instructions executable by the processor to perform:
receiving a query for at least one chunk of data stored in the server;
generating a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein the at least one data summary of the at least one chunk of data takes the form of a relational data table with columns and rows, wherein the at least one data summary comprises at least one histogram reflecting intensities of occurrences of values of at least one column, and at least one co-occurrence ratio reflecting intensities of co-occurrences of values of a plurality of columns; and
transmitting the generated response to the query.
17. The system of claim 16 , wherein the at least one histogram reflects intensities of occurrences within a quantized domain of values of the at least one column within the at least one data chunk, and wherein the at least one co-occurrence ratio reflects intensities of co-occurrences of quantized values of the plurality of columns.
18. The system of claim 17 , wherein the domain of values of the at least one column is quantized based on heuristic classification of the estimated expected accuracy of the generated response to the query subject to different choices of quantization settings.
19. The system of claim 17 , wherein the at least one co-occurrence ratio does not represent information about all possible combinations of values of the plurality of columns and the co-occurrence ratios related to unrepresented combinations of values are estimated based on information about the at least one co-occurrence ratio related to combinations of the quantized values.
20. The system of claim 17 , wherein the at least one co-occurrence ratio to be represented is selected from a set of all combinations of values of the plurality of columns based on heuristic classification of the estimated expected accuracy of the generated response to the query.
21. A computer program product, for capture and transformation of granulated data summaries from a data set stored in a server, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform the method comprising:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for the at least one chunk of data comprises:
dividing, at the computer system, the data stored in the server into a plurality of chunks;
generating, at the computer system, statistical information for at least some columns in some of the chunks of the data;
generating, at the computer system, statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating, at the computer system, at least one data summary based on the generated statistical information for the at least some columns and the generated statistical information about relationships between at least some pairs of columns; and
transmitting, at the computer system, the generated response to the query.
22. The computer program product of claim 21 , wherein generating a response to the query comprises utilizing, at the computer system, a mechanism of tree-based belief propagation adapted to populate changes in the at least one data summary.
23. The computer program product of claim 21 , wherein the generated response to the query is a standard SQL result.
24. A computer program product, for capture and transformation of granulated data summaries from a data set stored in a server, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform the method comprising:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein generating at least one data summary for the at least one chunk of data comprises:
receiving, at the computer system, data including a plurality of rows of the data at the computer system;
generating, at the computer system, at least a chunk of the data including a predetermined number of received rows of the data;
generating, at the computer system, statistical information for the generated chunks of the data;
generating, at the computer system, statistical information about relationships between at least some pairs of columns of the chunks of data; and
generating, at the computer system, at least one data summary based on the generated statistical information for the generated chunks of data and on the generated statistical information about relationships between the at least some pairs of columns of the chunks of data; and
transmitting, at the computer system, the generated response to the query.
25. The computer program product of claim 24 , wherein the statistical information for the generated chunks of data and the statistical information about relationships are updated using the at least one data summary that is generated as data is received by the computer system.
26. A computer program product, for capture and transformation of granulated data summaries from a data set stored in a server, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:
receiving, at the computer system, a query for at least one chunk of data stored in the server;
generating, at the computer system, a response to the query including a result generated from at least one data summary of the at least one chunk of data without accessing the data itself, wherein the at least one data summary of the at least one chunk of data takes the form of a relational data table with columns and rows, wherein the at least one data summary comprises at least one histogram reflecting intensities of occurrences of values of at least one column, and at least one co-occurrence ratio reflecting intensities of co-occurrences of values of a plurality of columns; and
transmitting, at the computer system, the generated response to the query.
27. The computer program product of claim 26 , wherein the at least one histogram reflects intensities of occurrences within a quantized domain of values of the at least one column within the at least one data chunk, and wherein the at least one co-occurrence ratio reflects intensities of co-occurrences of quantized values of the plurality of columns.
28. The computer program product of claim 27 , wherein the domain of values of the at least one column is quantized based on heuristic classification of the estimated expected accuracy of the generated response to the query subject to different choices of quantization settings.
29. The computer program product of claim 27 , wherein the at least one co-occurrence ratio does not represent information about all possible combinations of values of the plurality of columns and the co-occurrence ratios related to unrepresented combinations of values are estimated based on information about the at least one co-occurrence ratio related to combinations of the quantized values.
30. The computer program product of claim 27 , wherein the at least one co-occurrence ratio to be represented is selected from a set of all combinations of values of the plurality of columns based on heuristic classification of the estimated expected accuracy of the generated response to the query.