Method and system for handling large data sets in a statistical language
Methods and systems for providing support for large data sets are provided. Example embodiments provide a Big Data Object Library “BDOL,” which defines data structures and routines for handling big data objects using out of memory techniques. In one embodiment, the BDOL defines a bdFrame object which stores the data in binary form in a cache on an external storage medium, such as a file on a disk. The example BDOL provides support for user defined block processing a bdFrames using a pipeline engine. Also, the BDOL provides for Trellis plots, and other charts, of big data objects using hexagonal binning. This abstract is provided to comply with rules requiring an abstract, and it is submitted with the intention that it will not be used to interpret or limit the scope or meaning of the claims.
1. A method in a computing system for performing statistical computations in a statistical programming language on a data set that is larger than can fit in memory practicably, the data set comprising at least one row and at least one column, comprising:
inputting data from the data set into an external data cache area on an external storage medium, the external data cache area separate from virtual memory of the computing system;
under control of an interpreter of the statistical programming language, generating an in-memory data structure that contains an indicator to the external data cache area; and
under control of an interpreter of the statistical programming language, performing in memory a statistical computation, defined using the statistical programming language, on the inputted data, the data larger than can fit into physical and, when present, virtual memory practicably at once, by accessing and processing the data from the external data cache area a portion at a time through the generated in-memory structure.
2. The method of claim 1 , further comprising:
computing summary statistics on each column of the data set as it is input; and
storing the computed summary statistics in a metadata file on an external storage medium.
3. The method of claim 2 wherein the metadata file is associated with the in-memory data structure, such that statistical information can be provided regarding the data set without accessing the data stored in the external data cache.
4. The method of claim 2 wherein the summary statistics include at least one of a mean value, a minimum value, a maximum value, or a standard deviation of the values of the column data.
5. The method of claim 2 wherein the summary statistics provide a count of number of occurrences of a categorical value.
6. The method of claim 1 wherein the data stored in the external data cache is binary data.
7. The method of claim 1 wherein the statistical computation is performed using S-PLUS language.
8. The method of claim 1 wherein the statistical computation is a user-defined function that processes input data a block at a time.
9. The method of claim 8 wherein the block is a row.
10. The method of claim 1 wherein the portion of data accessed and processed at a time is at least one of a row, a determined number of rows, and/or is configurable.
11. The method of claim 1 wherein the performing the statistical computation is at least one of performing a data cleaning operation, a data manipulation operation, a data transformation operation, a data modeling operation, or a data visualization operation.
12. The method of claim 1 , further comprising:
presenting the processed data using hexagonal binning techniques.
13. The method of claim 1 , further comprising:
presenting a Trellis plot of the processed data using hexagonal binning techniques.
14. The method of claim 1 wherein the performing the statistical computation on the data further comprises:
performing a statistical computation on the data by using a pipeline engine to access and process the data a portion at a time.
15. The method of claim 1 wherein the statistical computation is at least one of performing a linear regression, a generalized linear regression, clustering, or principal components analysis.
16. A computer-readable memory medium whose contents, when executed, cause a computing device to handle a large data set for statistical processing, by performing a method comprising:
inputting data from the data set into an external data cache area on an external storage medium, the external data cache area separate from virtual memory of the computing device;
generating an in-memory data structure that contains an indicator to the external data cache area; and
performing in memory a statistical computation, defined using a statistical programming language, on the inputted data, the data larger than can fit into physical and, when present, virtual memory practicably at once, by accessing and processing the data from the external data cache area a portion at a time through the generated in-memory structure.
17. The computer-readable medium of claim 16 wherein the computer-readable memory medium is a memory of a computing device.
18. The computer-readable memory medium of claim 16 wherein the contents are instructions that, when executed, cause the computing device to perform the method.
19. A computing system, comprising:
a random access memory (“RAM”);
an external storage medium;
a code module stored in the computing system, configured to:
instantiate an object in RAM that points to an external data cache stored on the external storage medium, the external data cache separate from any virtual memory of the computing system;
retrieve, from a data set, data that is larger than can be stored in the RAM and, when present, virtual memory, without causing the computing system to thrash, and store the retrieved data in the external data cache; and
provide a function for accessing and processing data stored in the external data cache a block at a time; and
a statistical component defined in a statistical programming language that is configured to:
invoke the code module to instantiated the object, input data from an external source, and store the inputted data in the external data cache;
retrieve the stored data from the external data cache through the instantiated object and perform statistical modeling of the data using out-of-memory techniques that invoke the provided function to access and model the data a block at a time from the external data cache.
20. The computing system of claim 19 wherein the function of the code module for processing data stored in the external cache operates through a pipeline engine.
21. The computing system of claim 19 wherein the statistical modeling provides at least one of a linear regression; a generalized linear regression, clustering, or a principal components analysis of the data stored in the external cache.
22. The computing system of claim 19 , further comprising:
a graphical presentation component that provides a visualization of the data stored in the external cache using hexagonal binning techniques.
23. The computing system of claim 22 wherein the graphical presentation component provides the visualization using Trellis plots.