Accelerated automatic creation of data transformations

1 . A computer-implemented method comprising:

receiving, from a user computer at a server computer, a configuration file specifying two or more data transformations and associated parameters for executing the two or more data transformations;

the server computer initiating execution of a data transformation framework by loading one or more configuration parameters of the configuration file into main memory of the server computer to define a workflow;

creating and storing a configuration table based on the configuration file, the configuration table comprising a plurality of dynamic queries, a plurality of identifiers of transformation functions, and names of the two or more data transformations;

in the configuration table, for each of the two or more data transformations, creating at least one dynamic common table expression query, the at least one dynamic common table expression query being associated with a particular transformation function matching one of the plurality of identifiers of the transformation functions;

based on the configuration table, executing the at least one dynamic common table expression query to serially execute the two or more data transformations.

2 . The method of claim 1 , each transformation among the two or more data transformations comprising one of deduplication, cleaning, standardization, lookup, join, sub procedure, filter, hashing, parsing, matrix operations, pivoting, or transposition.

3 . The method of claim 1 , further comprising deleting one or more intermediate tables or staging tables that have been created as a result of executing the at least one dynamic common table expression query.

4 . The method of claim 1 , the configuration file specifying, for each of the two or more data transformations, a data processing engine, and a plurality of operating parameters specific to the data processing engine; the method further comprising executing the at least one dynamic common table expression query in part by invoking each data processing engine specified in the configuration file using the plurality of operating parameters specific to the data processing engine.

5 . The method of claim 1 , the configuration file specifying, for one or more of the two or more data transformations, a specification of a nested sub process prior to a join transform, the sub process specifying one or more data source identifiers of source database tables in a target database system, one or more data sink identifiers of sink database tables in the target database system, and two or more additional data transformations; the method further comprising executing the at least one dynamic common table expression query in part by first executing the two or more additional data transformations specified in the sub process followed by a join of one of the sink database tables.

6 . The method of claim 5 , the configuration file specifying at least one join transform that references, as a dimensional source table, one of the sink database tables of the nested sub process.

7 . The method of claim 1 , the at least one dynamic common table expression query comprising at least one HIVE query and at least one PRESTO query.

8 . The method of claim 1 , further comprising:

specifying one or more data source identifiers of source database tables in a target database system and one or more data sink identifiers of sink database tables in the target database system; and

initiating execution of the data transformation framework by invoking a DIGDAG orchestration file that specifies including a database configuration, including a specification of source parameters, and establishes parallel execution of processing of each of the data source identifiers, the parallel execution comprising, for each of the source database tables, executing a staging query, building a plurality of transforms based on the two or more data transformations including nesting one or more sub processes, and running the plurality of transforms.

9 . The method of claim 1 , wherein the configuration file specifies a list of the data transformations in an order in which they are configured to run.

10 . The method of claim 1 , wherein the two or more data transformations are specified in terms of a type, wherein the type is either full or incremental, and wherein the incremental type specifies that a data transformation executes deduplication last and is being run on incremental data being stored in one or more intermediate tables.

11 . A computer system comprising:

one or more processors;

one or more non-transitory computer-readable data storage media communicatively coupled to the one or more processors and storing one or more sequences of stored program instructions which, when executed using the one or more processors, cause the one or more processors to execute:

receiving, from a user computer at a server computer, a configuration file specifying two or more data transformations and associated parameters for executing the two or more data transformations;

the server computer initiating execution of a data transformation framework by loading one or more configuration parameters of the configuration file into main memory of the server computer to define a workflow;

creating and storing a configuration table based on the configuration file, the configuration table comprising a plurality of dynamic queries, a plurality of identifiers of transformation functions, and names of the two or more data transformations;

in the configuration table, for each of the two or more data transformations, creating at least one dynamic common table expression query, the at least one dynamic common table expression query being associated with a particular transformation function matching one of the plurality of identifiers of the transformation functions;

based on the configuration table, executing the at least one dynamic common table expression query to serially execute the two or more data transformations.

12 . The computer system of claim 11 , each transformation among the two or more data transformations comprising one of deduplication, cleaning, standardization, lookup, join, sub procedure, filter, hashing, parsing, matrix operations, pivoting, or transposition.

13 . The computer system of claim 11 , further comprising sequences of stored program instructions which, when executed using the one or more processors, cause the one or more processors to execute deleting one or more intermediate tables or staging tables that have been created as a result of executing the at least one dynamic common table expression query.

14 . The computer system of claim 11 , the configuration file specifying, for each of the two or more data transformations, a data processing engine, and a plurality of operating parameters specific to the data processing engine;

the system further comprising sequences of stored program instructions which, when executed using the one or more processors, cause the one or more processors to execute the at least one dynamic common table expression query in part by invoking each data processing engine specified in the configuration file using the plurality of operating parameters specific to the data processing engine.

15 . The computer system of claim 11 , the configuration file specifying, for one or more of the two or more data transformations, a specification of a nested sub process prior to a join transform, the sub process specifying one or more data source identifiers of source database tables in a target database system, one or more data sink identifiers of sink database tables in the target database system, and two or more additional data transformations;

the system further comprising sequences of stored program instructions which, when executed using the one or more processors, cause the one or more processors to execute the at least one dynamic common table expression query in part by first executing the two or more additional data transformations specified in the sub process followed by a join of one of the sink database tables.

16 . The computer system of claim 15 , the configuration file specifying at least one join transform that references, as a dimensional source table, one of the sink database tables of the nested sub process.

17 . The computer system of claim 11 , the at least one dynamic common table expression query comprising at least one HIVE query and at least one PRESTO query.

18 . The computer system of claim 11 , further comprising sequences of stored program instructions which, when executed using the one or more processors, cause the one or more processors to:

specify one or more data source identifiers of source database tables in a target database system and one or more data sink identifiers of sink database tables in the target database system; and

initiate execution of the data transformation framework by invoking a DIGDAG orchestration file that specifies including a database configuration, including a specification of source parameters, and establishes parallel execution of processing of each of the data source identifiers, the parallel execution comprising, for each of the source database tables, executing a staging query, building a plurality of transforms based on the two or more data transformations including nesting one or more sub processes, and running the plurality of transforms.

19 . The computer system of claim 11 , wherein the configuration file specifies a list of the data transformations in an order in which they are configured to run.

20 . The computer system of claim 11 , wherein the two or more data transformations are specified in terms of a type, wherein the type is either full or incremental, and wherein the incremental type specifies that a data transformation executes deduplication last and is being run on incremental data being stored in one or more intermediate tables.