Cache optimization for data preparation
Cache optimization for data preparation includes generating a data traversal program that represents a result of a set of sequenced data preparation operations performed on one or more sets of data. The data traversal program indicates how to assemble one or more affected columns in the one or more sets of data to derive the result. It further includes in response to receiving a specification of the set of sequenced operations to be performed on the one or more sets of data, accessing the data traversal program that represents the result or a stored copy of the data traversal program that represents the result. It further includes assembling the one or more affected columns in the one or more sets of data according to the data traversal program to re-generate the result. It further includes outputting the result.
1. A system, comprising:
a processor configured to:
while a first plurality of sequenced data preparation operations is sequentially performed on one or more input data sets, generate a data traversal program that represents a cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets, wherein the data traversal program comprises:
a references table comprising references to mappings of row transformations that occurred during the performing of the first plurality of sequenced data preparation operations; and
a reference stack comprising references to one or more file sets that store column files of data values that were written based at least in part on the performing of the first plurality of sequenced data preparation operations;
cache the generated data traversal program representing the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
at a time subsequent to caching the generated data traversal program, receive, via a user interface, a specification of a second plurality of sequenced data preparation operations;
in response to determining that at least a portion of the second plurality of sequenced data preparation operations matches at least a portion of the first plurality of sequenced data preparation operations, access the cached data traversal program that represents the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
re-generate the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets at least in part by executing the cached data traversal program, the executing of the data traversal program comprising using the reference stack in conjunction with the references table to obtain a set of rows that are at least a subset of a data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets; and
output the re-generated cumulative result, wherein outputting the re-generated cumulative result comprises displaying in the user interface the obtained set of rows that are at least a subset of the data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 wherein the data traversal program requires a less amount of memory storage than the cumulative result itself.
3. The system of claim 1 wherein generating the data traversal program does not duplicate the one or more input data sets.
4. The system of claim 1 wherein data pertaining to the data traversal program is stored automatically.
5. The system of claim 1 wherein data pertaining to the data traversal program is stored based on a complexity of the first plurality of sequenced data preparation operations.
6. The system of claim 1 wherein data pertaining to the data traversal program is stored in response to a request to store the cumulative result.
7. The system of claim 1 wherein the obtained set of rows are displayed in a windowed view.
8. The system of claim 1 wherein the data traversal program is associated with a set of one or more signatures.
9. The system of claim 8 wherein each signature in the set of one or more signatures is generated using a hash function.
10. A method, comprising:
while a first plurality of sequenced data preparation operations is sequentially performed on one or more input data sets, generating a data traversal program that represents a cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets, wherein the data traversal program comprises:
a references table comprising references to mappings of row transformations that occurred during the performing of the first plurality of sequenced data preparation operations; and
a reference stack comprising references to one or more file sets that store column files of data values that were written based at least in part on the performing of the first plurality of sequenced data preparation operations;
caching the generated data traversal program representing the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
at a time subsequent to caching the generated data traversal program, receiving, via a user interface, a specification of a second plurality of sequenced data preparation operations;
in response to determining that at least a portion of the second plurality of sequenced data preparation operations matches at least a portion of the first plurality of sequenced data preparation operations, accessing the cached data traversal program that represents the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
re-generating the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets at least in part by executing the cached data traversal program, the executing of the data traversal program comprising using the reference stack in conjunction with the references table to obtain a set of rows that are at least a subset of a data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets; and
outputting the re-generated cumulative result, wherein outputting the re-generated cumulative result comprises displaying in the user interface the obtained set of rows that are at least a subset of the data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets.
11. The method of claim 10 wherein the data traversal program requires a less amount of memory storage than the cumulative result itself.
12. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
while a first plurality of sequenced data preparation operations is sequentially performed on one or more input data sets, generating a data traversal program that represents a cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets, wherein the data traversal program comprises:
a references table comprising references to mappings of row transformations that occurred during the performing of the first plurality of sequenced data preparation operations; and
a reference stack comprising references to one or more file sets that store column files of data values that were written based at least in part on the performing of the first plurality of sequenced data preparation operations;
caching the generated data traversal program representing the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
at a time subsequent to caching the generated data traversal program, receiving, via a user interface, a specification of a second plurality of sequenced data preparation operations;
in response to determining that at least a portion of the second plurality of sequenced data preparation operations matches at least a portion of the first plurality of sequenced data preparation operations, accessing the cached data traversal program that represents the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets;
re-generating the cumulative result of sequentially performing the first plurality of sequenced data preparation operations on the one or more input data sets at least in part by executing the cached data traversal program, the executing of the cached data traversal program comprising using the reference stack in conjunction with the references table to obtain a set of rows that are at least a subset of a data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets; and
outputting the re-generated cumulative result, wherein outputting the re-generated cumulative result comprises displaying in the user interface the obtained set of rows that are at least a subset of the data set resulting from the first plurality of sequenced data preparation operations having been performed on the one or more input data sets.