Operator approach for generic dataflow designs
Described herein is an operator-based approach to representing dataflows. A dataflow is a set of one or more operations and one or more flows of data that are processed successively by the set of operations. A dataflow is described by a generic description in which operations in a dataflow are represented by operators. An operator defines a primitive operation (e.g. join, filter), specifying not only the type of operation, but the inputs and outputs, rules, and criteria that govern the operation. From the generic description, a code implementation is generated that may be completely executed on a source database system and target data warehouse, without need for an intermediate system to participate in the execution of the code implementation, such as a data movement engine.
1. A method of generating code that implements a dataflow plan for extracting data from a source database and transforming said data, the method comprising the computer-implemented steps of:
storing a description of a dataflow that represents the dataflow plan using operators that process data flowing between the operators;
wherein each operator of the operators defines an operation which has an input rowset and an output rowset;
wherein the description defines, for each operator of the operators, a structure for the input rowset and a structure for the output rowset;
making a determination of which particular computer language from a plurality of computer languages in which to generate first code;
generating the first code in conformance with the particular computer language based on:
an evaluation of said dataflow plan, and
the determination of the particular computer languages;
wherein each operator of the operators corresponds to a different portion of the first code.
2. The method of claim 1 , wherein the step of generating the first code includes generating two subsets of code, each subset conforming to a different computer language.
3. The method of claim 1 , wherein the steps further include generating second code that implements said dataflow, said second code conforming to a second computer language different than the particular computer language.
4. The method of claim 1 , wherein the determination of the particular computer language is based on user input that specifies the particular computer language.
5. The method of claim 1 , wherein said description defines an input attribute group that defines the structure of the input rowset and an output attribute group that defines the structure of the output rowset.
6. The method of claim 5 , wherein the output attribute group and the input attribute group are the same attribute group.
7. The method of claim 6 , wherein said description defines that a field of a particular output rowset is a field of a particular input rowset of another operator by mapping an attribute of the output attribute group that defines the particular output rowset to an attribute of the input attribute group that defines the particular input rowset.
8. The method of claim 5 , further include generating binding data that binds attributes of one or more attribute groups defined by said description to entities in a database.
9. The method of claim 8 , wherein the step of generating first code includes generating code that specifies operations that operate on said entities.
10. The method of claim 1 , wherein:
said operators specify an operation; and
the step of generating code includes generating for each operator of said operators a generation context that represents a code implementation that implements the operation specified by the operator and the operation specified by one or more upstream operators.
11. The method of claim 10 , wherein:
the step of generating a generation context is based on examining a definition of said each operator and one or more generation contexts generated for the one or more upstream operators.
12. The method of claim 1 , wherein:
the steps include receiving user input specifying a deployment language, said deployment language being associated with at least one code generation strategy governing the behavior and characteristics of code to generate; and
the step of generating the first code is based on said code generation strategy.
13. The method of claim 1 , wherein the steps include displaying to the user intermediate results of said generating code based on user selection indicating an attribute group of said operators.
14. The method of claim 1 , wherein making the determination includes making the determination, without user input, based on which computer language, of the plurality of computer languages, is optimal for implementing the dataflow plan in a deployment environment in which the first code will be executed.
15. A computer-readable storage medium storing instructions for generating code that implements a dataflow plan for extracting data from a source database and transforming said data, wherein the instructions, when executed by one or more processors, cause the one or more processors to perform the steps of:
storing a description of a dataflow that represents the dataflow plan using operators that process data flowing between the operators;
wherein each operator of the operators defines an operation which has an input rowset and an output rowset;
wherein the description defines, for each operator of the operators, a structure for the input rowset and a structure for the output rowset;
making a determination of which particular computer language from a plurality of computer languages in which to generate first code;
generating the first code in conformance with the particular computer language based on: an evaluation of said dataflow plan, and
the determination of the particular computer language; wherein each operator of the operators corresponds to a different portion of the first code.
16. The computer-readable storage medium of claim 15 , wherein the step of generating the first code includes generating two subsets of code, each subset conforming to a different computer language.
17. The computer-readable storage medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of generating second code that implements said dataflow, said second code conforming to a second computer language different than the particular computer language.
18. The computer-readable storage medium of claim 15 , wherein the determination of the particular computer language is based on user input that specifies the particular computer language.
19. The computer-readable storage medium of claim 15 , wherein said description defines an input attribute group that defines the structure of the input rowset and an output attribute group that defines the structure of the output rowset.
20. The computer-readable storage medium of claim 19 , wherein the output attribute group and the input attribute group are the same attribute group.
21. The computer-readable storage medium of claim 20 , wherein said description defines that a field of a particular output rowset is a field of a particular input rowset of another operator by mapping an attribute of the output attribute group that defines the particular output rowset to an attribute of the input attribute group that defines the particular input rowset.
22. The computer-readable storage medium of claim 19 , further include generating binding data that binds attributes of one or more attribute groups defined by said description to entities in a database.
23. The computer-readable storage medium of claim 22 , wherein the step of generating first code includes generating code that specifies operations that operate on said entities.
24. The computer-readable storage medium of claim 15 , wherein:
said operators specify an operation; and
the step of generating code includes generating for each operator of said operators a generation context that represents a code implementation that implements the operation specified by the operator and the operation specified by one or more upstream operators.
25. The computer-readable storage medium of claim 24 , wherein:
the step of generating a generation context is based on examining a definition of said each operator and one or more generation contexts generated for the one or more upstream operators.
26. The computer-readable storage medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of:
receiving user input specifying a deployment language, said deployment language being associated with at least one code generation strategy governing the behavior and characteristics of code to generate;
wherein the step of generating the first code is based on said code generation strategy.
27. The computer-readable storage medium of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of displaying to the user intermediate results of said generating code based on user selection indicating an attribute group of said operators.
28. The computer-readable storage medium of claim 15 , wherein making the determination includes making the determination, without user input, based on which computer language, of the plurality of computer languages, is optimal for implementing the dataflow plan in a deployment environment in which the first code will be executed.