CACHE OPTIMIZATION FOR DATA PREPARATION
Cache optimization for data preparation includes: generating a data traversal program that represents a result of a set of sequenced data preparation operations performed on one or more sets of data, wherein the data traversal program indicates how to assemble one or more affected columns in the one or more sets of data to derive the result; in response to receiving a specification of the set of sequenced operations to be performed on the one or more sets of data, accessing the data traversal program that represents the result or a stored copy of the data traversal program that represents the result; assembling the one or more affected columns in the one or more sets of data according to the data traversal program to re-generate the result; and outputting the result.
1 - 20 . (canceled)
21 . A method, comprising:
establishing, by one or more processors coupled to memory, a plurality programs configured to execute a set of operations on a corresponding plurality of partitions of a set of data to generate one or more results;
receiving, by the one or more processors, a request to perform the set of operations on the set of data;
determining, by the one or more processors, that a first subset of the set of data is viewable via a graphical user interface displayed via a display device, and that a second subset of the set of data is hidden from view on the graphical user interface;
selecting, by the one or more processors, one or more programs of the plurality of programs that correspond to one or more partitions of the plurality of partitions that map to the first subset of the set of data that is viewable;
executing, by the one or more processors and in response to receiving the request, the selected one or more programs on the corresponding one or more partitions to generate a result for the first subset of the set of data that is viewable; and
providing, by the one or more processors, the result for the selected first subset of the set of data for display via the graphical user interface.
22 . The method of claim 21 , further comprising:
traversing, by the one or more processors, the set of data to generate a plurality of partial results, the result comprising the plurality of partial results.
23 . The method of claim 21 , wherein the data set comprises at least one table, and the subset comprises one or more cells of the table.
24 . The method of claim 23 , further comprising:
generating, by the one or more processors, the partitions based on one or more of a number of rows of the table and a size of corresponding sets of data contained by one or more of the rows of the table.
25 . The method of claim 24 , further comprising:
selecting, by the one or more processors, one or more columns of the table associated with the set of operations; and
generating, by the one or more processors, the partitions based on a size of corresponding sets of data contained by one or more selected columns, the selected columns selected from the columns of the table associated with the set of operations.
26 . The method of claim 21 , further comprising:
deriving, by the one or more processors, a first signature based on the set of operations; and
caching, by the one or more processors, the program.
27 . The method of claim 26 , further comprising:
receiving, by the one or more processors, a second request to perform a second set of operations on the set of data;
deriving, by the one or more processors, a second signature based on the second set of operations;
determining, by the one or more processors, whether the second signature matches the first signature;
obtaining, by the one or more processors, the cached program; and
executing, by the one or more processors and in response to receiving the second request, the second result for the first subset of the set of data that is viewable.
28 . The method of claim 26 , further comprising:
receiving, by the user interface, a save point identifying a subset of the set of operations,
wherein the first signature is based on the subset of the set of operations.
29 . The method of claim 21 , further comprising:
modifying, by the one or more processors, an order of one or more operations of the set of operations, in response to a determination that the modifying reduces computation associated with the set of operations.
30 . A system, comprising:
a data processing system comprising memory and one or more processors to:
establish a plurality programs configured to execute a set of operations on a corresponding plurality of partitions of a set of data to generate one or more results;
receive a request to perform the set of operations on the set of data;
determine that a first subset of the set of data is viewable via a graphical user interface displayed via a display device, and that a second subset of the set of data is hidden from view on the graphical user interface;
select one or more programs of the plurality of programs that correspond to one or more partitions of the plurality of partitions that map to the first subset of the set of data that is viewable;
execute, in response to receiving the request, the selected one or more programs on the corresponding one or more partitions to generate a result for the first subset of the set of data that is viewable; and
provide the result for the selected first subset of the set of data for display via the graphical user interface.
31 . The system of claim 30 , the data processing system further configured to:
traverse the set of data to generate a plurality of partial results, the result comprising the plurality of partial results.
32 . The system of claim 30 , wherein the data set comprises at least one table, and the subset comprises one or more cells of the table.
33 . The system of claim 32 , the data processing system further configured to:
generate the partitions based on one or more of a number of rows of the table and a size of corresponding sets of data contained by one or more of the rows of the table.
34 . The system of claim 33 , the data processing system further configured to:
select one or more columns of the table associated with the set of operations; and
generate the partitions based on a size of corresponding sets of data contained by one or more selected columns, the selected columns selected from one or more columns of the table associated with the set of operations.
35 . The system of claim 30 , the data processing system further configured to:
derive a first signature based on the set of operations; and
cache the program.
36 . The system of claim 35 , the data processing system further configured to:
receive a second request to perform a second set of operations on the set of data;
derive a second signature based on the second set of operations;
determine whether the second signature matches the first signature;
obtain the cached program; and
execute, by the one or more processors and in response to receiving the second request, the second result for the first subset of the set of data that is viewable.
37 . The system of claim 35 , the data processing system further configured to:
receive, by the user interface, a save point identifying a subset of the set of operations,
wherein the first signature is based on the subset of the set of operations.
38 . The system of claim 30 , the data processing system further configured to:
modify an order of one or more operations of the set of operations, in response to a determination that the modifying reduces computation associated with the set of operations.
39 . A computer readable medium including one or more instructions stored thereon and executable by a processor to:
establish a plurality programs configured to execute a set of operations on a corresponding plurality of partitions of a set of data to generate one or more results;
receive a request to perform the set of operations on the set of data;
determine that a first subset of the set of data is viewable via a graphical user interface displayed via a display device, and that a second subset of the set of data is hidden from view on the graphical user interface;
select one or more programs of the plurality of programs that correspond to one or more partitions of the plurality of partitions that map to the first subset of the set of data that is viewable;
execute, in response to receiving the request, the selected one or more programs on the corresponding one or more partitions to generate a result for the first subset of the set of data that is viewable; and
provide the result for the selected first subset of the set of data for display via the graphical user interface.
40 . The computer readable medium of claim 39 , wherein the computer readable medium further includes one or more instructions executable by the processor to:
derive a first signature based on the set of operations;
cache the program;
receive a second request to perform a second set of operations on the set of data;
derive a second signature based on the second set of operations;
determine whether the second signature matches the first signature;
obtain the cached program; and
execute, by the one or more processors and in response to receiving the second request, the second result for the first subset of the set of data that is viewable.