Creating virtualized data assets using existing definitions of ETL/ELT jobs
A computer-implemented method, system and computer program product for creating virtualized data assets. A copy of an extract, transform, load (ETL) or an extract, load, transform (ELT) job definition is created. Furthermore, a table function is created in which the table function is configured to trigger the execution of a copy of the ETL/ELT job definition, to read the resulting data from executing the copy of the ETL/ELT job definition and to return the resulting data (data sets), such as in a queue in a federation server. After receiving a query from a user based on a virtual view in which the query includes the ETL/ELT job definition, the table function is executed which triggers execution of the copy of the ETL/ELT job definition. Data sets are then produced by executing the copied ETL/ELT job definition, which is read and returned to the user in the virtual view.
1. A computer-implemented method for creating virtualized data assets, the method comprising:
receiving a set of extract, transform, load (ETL) or extract, load, transform (ELT) job definitions;
translating said set of ETL or ELT job definitions to statements in a domain-specific language;
creating virtualized data assets using said translated statements; and
representing a federated view for execution of said translated statements that includes said created virtualized data assets.
2. The method as recited in claim 1 further comprising:
building a first catalog to store said created virtualized data assets with corresponding ETL or ELT job definitions.
3. The method as recited in claim 2 further comprising:
creating a second set of ETL or ELT job definitions to use associated translated statements to be executed against said virtualized data assets of said first catalog.
4. The method as recited in claim 3 further comprising:
building a second catalog defining how data is extracted, transformed and loaded or how data is extracted, loaded and transformed (ELT) based on said set of ETL or ELT job definitions, respectively, wherein said second catalog is modified by replacing said set of ETL or ELT job definitions with said created second set of ETL or ELT job definitions.
5. The method as recited in claim 1 , wherein said domain-specific language comprises a structured query language.
6. The method as recited in claim 1 , wherein said translation of said set of ETL or ELT job definitions to statements in said domain-specific language is accomplished via a table of ETL or ELT stages with corresponding statements in said domain-specific language.
7. The method as recited in claim 1 further comprising:
storing said translated statements in a third catalog.
8. A computer program product for creating virtualized data assets, the computer program product comprising one or more computer readable storage mediums having program code embodied therewith, the program code comprising programming instructions for:
receiving a set of extract, transform, load (ETL) or extract, load, transform (ELT) job definitions;
translating said set of ETL or ELT job definitions to statements in a domain-specific language;
creating virtualized data assets using said translated statements; and
representing a federated view for execution of said translated statements that includes said created virtualized data assets.
9. The computer program product as recited in claim 8 , wherein the program code further comprises the programming instructions for:
building a first catalog to store said created virtualized data assets with corresponding ETL or ELT job definitions.
10. The computer program product as recited in claim 9 , wherein the program code further comprises the programming instructions for:
creating a second set of ETL or ELT job definitions to use associated translated statements to be executed against said virtualized data assets of said first catalog.
11. The computer program product as recited in claim 10 , wherein the program code further comprises the programming instructions for:
building a second catalog defining how data is extracted, transformed and loaded or how data is extracted, loaded and transformed (ELT) based on said set of ETL or ELT job definitions, respectively, wherein said second catalog is modified by replacing said set of ETL or ELT job definitions with said created second set of ETL or ELT job definitions.
12. The computer program product as recited in claim 8 , wherein said domain-specific language comprises a structured query language.
13. The computer program product as recited in claim 8 , wherein said translation of said set of ETL or ELT job definitions to statements in said domain-specific language is accomplished via a table of ETL or ELT stages with corresponding statements in said domain-specific language.
14. The computer program product as recited in claim 8 , wherein the program code further comprises the programming instructions for:
storing said translated statements in a third catalog.
15. A system, comprising:
a memory for storing a computer program for creating virtualized data assets; and
a processor connected to the memory, wherein the processor is configured to execute program instructions of the computer program comprising:
receiving a set of extract, transform, load (ETL) or extract, load, transform (ELT) job definitions;
translating said set of ETL or ELT job definitions to statements in a domain-specific language;
creating virtualized data assets using said translated statements; and
representing a federated view for execution of said translated statements that includes said created virtualized data assets.
16. The system as recited in claim 15 , wherein the program instructions of the computer program further comprise:
building a first catalog to store said created virtualized data assets with corresponding ETL or ELT job definitions.
17. The system as recited in claim 16 , wherein the program instructions of the computer program further comprise:
creating a second set of ETL or ELT job definitions to use associated translated statements to be executed against said virtualized data assets of said first catalog.
18. The system as recited in claim 17 , wherein the program instructions of the computer program further comprise:
building a second catalog defining how data is extracted, transformed and loaded or how data is extracted, loaded and transformed (ELT) based on said set of ETL or ELT job definitions, respectively, wherein said second catalog is modified by replacing said set of ETL or ELT job definitions with said created second set of ETL or ELT job definitions.
19. The system as recited in claim 15 , wherein said domain-specific language comprises a structured query language.
20. The system as recited in claim 15 , wherein said translation of said set of ETL or ELT job definitions to statements in said domain-specific language is accomplished via a table of ETL or ELT stages with corresponding statements in said domain-specific language.