Domain-specific language for dataset transformations
View Patent ↗Techniques related to a domain-specific language for dataset transformations are disclosed. A server computer may process a table definition composed in a domain-specific language. The table definition may include a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table. The sequence may include a customized transformation. A source dataset may be provided as input to an implementation of the customized transformation. An output dataset may be generated as a result of executing the implementation. An intermediate table may be generated based on performing at least one dataset transformation on a particular source table. A supplemental portion for the intermediate table may be generated based on performing the at least one dataset transformation on an appended portion of the particular source table. The target table may be generated based on combining the supplemental portion with the intermediate table.
1. A method comprising:
processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table, wherein the one or more dataset transformations comprises a customized transformation;
based on processing the customized transformation in the one or more dataset transformations:
obtaining an implementation of the customized transformation;
executing the implementation based on providing a source dataset as input to the implementation;
generating an output dataset as a result of executing the implementation;
wherein an intermediate table is generated based on performing at least one dataset transformation of the one or more dataset transformations on a particular source table of the one or more source tables;
wherein the one or more source tables comprise an appended portion of the particular source table;
wherein the target table is generated based on:
generating a supplemental portion for the intermediate table based on performing the at least one dataset transformation on the appended portion of the particular source table;
combining intermediate table with the supplemental portion for the intermediate table;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the one or more dataset transformations are performed in parallel.
3. The method of claim 1 , wherein the sequence of one or more dataset transformations is determined based on a graphical representation of the one or more dataset transformations.
4. The method of claim 3 , wherein the graphical representation is a directed acyclic graph.
5. The method of claim 3 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.
6. The method of claim 1 , wherein the target table is generated based on persisting the output dataset.
7. The method of claim 1 , wherein one or more references to the one or more source tables are stored at a server computer in such a manner that the one or more source tables remain uncopied to the server computer.
8. A system comprising:
one or more processors; and
one or more storage media storing instructions which, when executed by the one or more processors, cause:
processing a table definition composed in a domain-specific language, the table definition comprising a sequence of one or more dataset transformations to be performed on one or more source tables to generate a target table, wherein the one or more dataset transformations comprises a customized transformation;
based on processing the customized transformation in the one or more dataset transformations:
obtaining an implementation of the customized transformation;
executing the implementation based on providing a source dataset as input to the implementation;
generating an output dataset as a result of executing the implementation;
wherein an intermediate table is generated based on performing at least one dataset transformation of the one or more dataset transformations on a particular source table of the one or more source tables;
wherein the one or more source tables comprise an appended portion of the particular source table;
wherein the target table is generated based on:
generating a supplemental portion for the intermediate table based on performing the at least one dataset transformation on the appended portion of the particular source table;
combining intermediate table with the supplemental portion for the intermediate table.
9. The system of claim 8 , wherein the one or more dataset transformations are performed in parallel.
10. The system of claim 8 , wherein the sequence of one or more dataset transformations is determined based on a graphical representation of the one or more dataset transformations.
11. The system of claim 10 , wherein the graphical representation is a directed acyclic graph.
12. The system of claim 10 , wherein the graphical representation comprises one or more leaf nodes and one or more non-leaf nodes, the one or more leaf nodes representing one or more tables, the one or more non-leaf nodes representing the one or more dataset transformations.
13. The system of claim 8 , wherein the target table is generated based on persisting the output dataset.
14. The system of claim 8 , wherein one or more references to the one or more source tables are stored at a server computer in such a manner that the one or more source tables remain uncopied to the server computer.