Domain-specific language for dataset transformations
Techniques related to a domain-specific language for transformations are disclosed. A server computer may process a table definition composed in a domain-specific language. The table definition may include a sequence of one or more transformations to be performed on one or more source tables to generate a target table. The sequence may include a customized transformation. A source dataset may be provided as input to an implementation of the customized transformation. An output dataset may be generated as a result of executing the implementation. An intermediate table may be generated based on performing at least one transformation on a particular source table. A supplemental portion for the intermediate table may be generated based on performing the at least one transformation on an appended portion of the particular source table. The target table may be generated based on combining the supplemental portion with the intermediate table.
1. A data processing method comprising:
retrieving an intermediate table that was generated based in part on performing a first transformation of one or more transformations on a version of a particular source table, wherein the one or more transformations are performable on one or more source tables to generate a target table;
generating a supplemental portion for the intermediate table based in part on performing a second transformation on an appended portion of an updated version of the particular source table;
generating the target table based in part on performing a third transformation on the intermediate table and the supplemental portion for the intermediate table,
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the first transformation and the second transformation are each dataset transformations and are different dataset transformations.
3. The method of claim 1 , wherein the second transformation and the third transformation are performed without being specified by an end user.
4. The method of claim 1 , wherein generating the supplemental portion comprises performing a fourth transformation on an output dataset of the first transformation and an output dataset of the second transformation.
5. The method of claim 4 , wherein the fourth transformation is reversible.
6. The method of claim 1 , wherein the target table is generated based on persisting an output dataset of the third transformation.
7. The method of claim 1 , wherein a sequence of one or more transformations is determined based on a graphical representation of the one or more transformations.
8. The method of claim 7 , wherein the graphical representation is a directed acyclic graph.
9. The method of claim 7 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more transformations.
10. The method of claim 1 , wherein one or more references to the one or more source tables are stored at a server computer, without copies of the one or more source tables being stored at the server computer.
11. A computer system comprising:
one or more processors;
one or more non-transitory data storage media coupled to the one or more processors and storing instructions which, when executed by the one or more processors, cause:
retrieving an intermediate table that was generated based on performing a first transformation of one or more transformations on a version of a particular source table, wherein the one or more transformations are performable on one or more source tables to generate a target table;
generating a supplemental portion for the intermediate table based in part on performing a second transformation on an appended portion of an updated version of the particular source table;
generating the target table based in part on performing a third transformation on the intermediate table and the supplemental portion for the intermediate table.
12. The system of claim 11 , wherein the first transformation and the second transformation are each dataset transformations and are different dataset transformations.
13. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause performing the second transformation and the third transformation without being specified by an end user.
14. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause performing generating the supplemental portion by performing a fourth transformation on an output dataset of the first transformation and an output dataset of the second transformation.
15. The system of claim 14 , wherein the fourth transformation is reversible.
16. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause generating the target table based on persisting an output dataset of the third transformation.
17. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause determining a sequence of one or more transformations based on a graphical representation of the one or more transformations.
18. The system of claim 17 , wherein the graphical representation is a directed acyclic graph.
19. The system of claim 17 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more transformations.
20. The system of claim 11 , wherein the one or more storage media further comprise sequences of instructions which when executed cause storing one or more references to the one or more source tables at a server computer, without corresponding copies of the one or more source tables being stored at the server computer.