Step editor for data preparation
Using a step editor for data preparation includes receiving an indication of a user input with respect to at least some of a set of sequenced data preparation operations on a set of data. It further includes generating, using one or more processors, a signature based at least in part on the set of sequenced data preparation operations, references to the set of data, and the user input. It further includes using the generated signature to determine whether there exists a cached result associated with the set of sequenced data preparation operations, the references to the set of data, and the user input. It further includes based at least in part on the determination, obtaining a data traversal program representing a result associated with the set of sequenced operations, the references to the set of data, and the user input. It further includes providing output based at least in part on the result represented by the obtained data traversal program.
1. A system, comprising:
one or more processors configured to:
display, via a user interface, a first set of sequenced data preparation operations to be sequentially performed on a set of data, the first set of sequenced data preparation operations displayed via the user interface comprising at least one of: clustering, joining, appending, sorting, uppercase, lowercase, filtering, deduplicating, grouping, adding or removing columns, adding or removing rows, pivoting, and/or depivoting;
receive, via the user interface, an indication of a user input, the user input received via the user interface comprising an indication to mute a data preparation operation in the first set of sequenced data preparation operations displayed via the user interface;
based at least in part on the user input comprising the indication to mute the data preparation operation in the first set of sequenced data preparation operations displayed via the user interface, determine a second set of sequenced data preparation operations, wherein the second set of sequenced data preparation operations comprises a version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
generate a signature based at least in part on the second set of sequenced data preparation operations comprising the version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
use the generated signature to determine that there exists a cached result matching to at least a portion of the second set of sequenced data preparation operations;
based at least in part on determining that there exists the cached result matching to the at least portion of the second set of sequenced data preparation operations, retrieve a data traversal program representing the cached result; and
provide, via the user interface, output based at least in part on the cached result represented by the retrieved data traversal program; and
a memory coupled to the one or more processors and configured to provide the one or more processors with instructions.
2. The system of claim 1 wherein the one or more processors are further configured to receive a user input to delete a step in the first set of sequenced data preparation operations.
3. The system of claim 1 wherein the one or more processors are further configured to receive a user input comprising an indication to step backwards through the first set of sequenced data preparation operations.
4. The system of claim 1 wherein the one or more processors are further configured to receive a user input comprising an indication to step forwards through the first set of sequenced data preparation operations.
5. The system of claim 1 wherein the signature comprises a hash based at least in part on the second set of sequenced data preparation operations.
6. The system of claim 1 wherein the data traversal program indicates how to assemble one or more affected columns in one or more sets of data to derive the cached result represented by the data traversal program.
7. The system of claim 1 wherein providing the output includes causing the output to be displayed.
8. A method, comprising:
displaying, via a user interface, a first set of sequenced data preparation operations to be sequentially performed on a set of data, the first set of sequenced data preparation operations displayed via the user interface comprising at least one of: clustering, joining, appending, sorting, uppercase, lowercase, filtering, deduplicating, grouping, adding or removing columns, adding or removing rows, pivoting, and/or depivoting;
receiving, via the user interface, an indication of a user input, the user input received via the user interface comprising an indication to mute a data preparation operation in the first set of sequenced data preparation operations displayed via the user interface;
based at least in part on the user input comprising the indication to mute the data preparation operation in the first set of sequenced data preparation operations displayed via the user interface, determining a second set of sequenced data preparation operations, wherein the second set of sequenced data preparation operations comprises a version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
generating, using one or more processors, a signature based at least in part on the second set of sequenced data preparation operations comprising the version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
using the generated signature to determine that there exists a cached result matching to at least a portion of the second set of sequenced data preparation operations;
based at least in part on determining that there exists the cached result matching to the at least portion of the second set of sequenced data preparation operations, retrieving a data traversal program representing the cached result; and
providing, via the user interface, output based at least in part on the cached result represented by the retrieved data traversal program.
9. The method of claim 8 further comprising receiving a user input comprising an indication to delete a step in the first set of sequenced data preparation operations.
10. The method of claim 8 further comprising receiving a user input comprising an indication to step backwards through the first set of sequenced data preparation operations.
11. The method of claim 8 further comprising receiving a user input comprising an indication to step forwards through the first set of sequenced data preparation operations.
12. A computer program product embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
displaying, via a user interface, a first set of sequenced data preparation operations to be sequentially performed on a set of data, the first set of sequenced data preparation operations displayed via the user interface comprising at least one of: clustering, joining, appending, sorting, uppercase, lowercase, filtering, deduplicating, grouping, adding or removing columns, adding or removing rows, pivoting, and/or depivoting;
receiving, via the user interface, an indication of a user input, the user input received via the user interface comprising an indication to mute a data preparation operation in the first set of sequenced data preparation operations displayed via the user interface;
based at least in part on the user input comprising the indication to mute the data preparation operation in the first set of sequenced data preparation operations displayed via the user interface, determining a second set of sequenced data preparation operations, wherein the second set of sequenced data preparation operations comprises a version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
generating a signature based at least in part on the second set of sequenced data preparation operations comprising the version of the first set of sequenced data preparation operations in which the muted data preparation operation is excluded;
using the generated signature to determine that there exists a cached result matching to at least a portion of the second set of sequenced data preparation operations;
based at least in part on determining that there exists the cached result matching to the at least portion of the second set of sequenced data preparation operations, retrieving a data traversal program representing the cached result; and
providing, via the user interface, output based at least in part on the cached result represented by the retrieved data traversal program.