Systems and methods for building an extract, transform, load pipeline
The disclosed computer-implemented method for building an extract, transform, load pipeline may include (i) identifying a plurality of ETL resources available to a user, (ii) categorizing each of the plurality of ETL resources based on at least one characteristic, (iii) provisioning the plurality of ETL resources for use with containers, (iv) presenting a user environment to the user, the user environment including a plurality of container images corresponding to available ETL resources, (v) receiving, from the user, a selection of at least one container image of the plurality of container images, and (vi) running at least one container from the at least one container image, the at least one container utilizing two or more ETL resources of the plurality of ETL resources. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for building an extract, transform, load (ETL) pipeline, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying a plurality of ETL resources employable in the ETL pipeline that are available to a user;
categorizing each of the plurality of ETL resources based on at least one characteristic of each of the plurality of ETL resources that affects a capability of the ETL pipeline when employed in the ETL pipeline;
provisioning the plurality of ETL resources for use with containers based on the at least one characteristic of each of the plurality of ETL resources;
presenting a user environment to the user, the user environment including a plurality of container images corresponding to available ETL resources of the plurality of ETL resources;
receiving, from the user, a selection of at least one container image of the plurality of container images, wherein the at least one container image includes at least one application that performs at least one action to be carried out in the ETL pipeline;
determining two or more ETL resources from the plurality of ETL resources that are best fit to carry out the at least one action based on the at least one characteristic of each of the plurality of ETL resources; and
running each of at least one container from the at least one container image that includes files and parameters for running the at least one container, the at least one container utilizing the two or more ETL resources of the plurality of ETL resources, wherein at least one of the two or more ETL resources comprises at least one server that hosts the at least one container, the at least one container isolating user space of the at least one container from other processes while sharing kernel space with the other processes on the at least one server.
2. The computer-implemented method of claim 1 , wherein the plurality of ETL resources comprise two or more of:
a server;
a database;
a network; and
a system.
3. The computer-implemented method of claim 1 , wherein the at least one characteristic of each of the plurality of ETL resources comprises at least one of:
input/output bandwidth;
location;
processing speed;
software compatibility; and
available storage space.
4. The computer-implemented method of claim 1 , wherein provisioning the plurality of ETL resources comprises provisioning the ETL resources with a container-based provisioner.
5. The computer-implemented method of claim 1 , further comprising grouping two or more of the plurality of ETL resources based on the at least one characteristic of the two or more of the plurality of ETL resources.
6. The computer-implemented method of claim 1 , further comprising obtaining at least one of the plurality of container images from an image database.
7. The computer-implemented method of claim 1 , further comprising creating at least one of the plurality of container images.
8. The computer-implemented method of claim 1 , wherein the user environment further shows a plurality of actions to be carried out with respect to the plurality of ETL resources, wherein the plurality of actions comprises the at least one action.
9. The computer-implemented method of claim 8 , wherein the plurality of actions comprises one or more actions to be carried out in each of a plurality of containers respectively corresponding to each of the plurality of container images.
10. The computer-implemented method of claim 8 , further comprising receiving, from the user, a selection of the at least one action of the plurality of actions to be carried out in a container corresponding to the at least one container image.
11. The computer-implemented method of claim 8 , wherein the plurality of actions comprises at least one of:
extracting data from a data source;
transforming extracted data; and
loading transformed data into a target data storage location.
12. The computer-implemented method of claim 11 , wherein transforming the extracted data comprises at least one of;
reformatting the extracted data;
standardizing the extracted data;
cleansing the extracted data; and
aggregating the extracted data.
13. The computer-implemented method of claim 1 , wherein the user environment further shows a set of ETL resources of the plurality of ETL resources.
14. The computer-implemented method of claim 13 , wherein the set of ETL resources comprises the available ETL resources of the plurality of ETL resources.
15. The computer-implemented method of claim 13 , wherein the set of ETL resources comprises ETL resources of the plurality of ETL resources that are determined to best fit requirements of at least one action to be carried out by the at least one container.
16. The computer-implemented method of claim 13 , wherein the set of ETL resources comprises at least one ETL resource from each of a plurality of groups of ETL resources.
17. The computer-implemented method of claim 13 , further comprising receiving, from the user, a selection of at least one of the two or more ETL resources utilized by the at least one container.
18. The computer-implemented method of claim 1 , wherein the user environment includes a graphical representation of the ETL pipeline including the two or more ETL resources and the at least one container.
19. A system for building an ETL pipeline, the system comprising:
an identifying module, stored in memory, that identifies a plurality of ETL resources employable in the ETL pipeline that are available to a user;
a categorizing module, stored in memory, that categorizes each of the plurality of ETL resources based on at least one characteristic of each of the plurality of ETL resources that affects a capability of the ETL pipeline when employed in the ETL pipeline;
a provisioning module, stored in memory, that provisions the plurality of ETL resources for use with containers based on the at least one characteristic of each of the plurality of ETL resources;
an interface module, stored in memory, that presents a user environment to the user, the user environment including a plurality of container images corresponding to available ETL resources of the plurality of ETL resources;
a receiving module, stored in memory, that receives, from the user, a selection of at least one container image of the plurality of container images, wherein the at least one container image includes at least one application that performs at least one action to be carried out in the ETL pipeline;
a determining module, stored in memory, that determines two or more ETL resources from the plurality of ETL resources that are best fit to carry out the at least one action based on the at least one characteristic of each of the plurality of ETL resources;
a running module, stored in memory, that runs each of at least one container from the at least one container image that includes files and parameters for running the at least one container, the at least one container utilizing two or more ETL resources of the plurality of ETL resources, wherein at least one of the two or more ETL resources comprises at least one server that hosts the at least one container, the at least one container isolating user space of the at least one container from other processes while sharing kernel space with the other processes on the at least one server; and
at least one processor that executes the identifying module, the categorizing module, the provisioning module, the interface module, the receiving module, the determining module, and the running module.
20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify a plurality of ETL resources employable in an ETL pipeline that are available to a user;
categorize each of the plurality of ETL resources based on at least one characteristic of each of the plurality of ETL resources that affects a capability of the ETL pipeline when employed in the ETL pipeline;
provision the plurality of ETL resources for use with containers based on the at least one characteristic of each of the plurality of ETL resources;
present a user environment to the user, the user environment including a plurality of container images corresponding to available ETL resources of the plurality of ETL resources;
receive, from the user, a selection of at least one container image of the plurality of container images, wherein the at least one container image includes at least one application that performs at least one action to be carried out in the ETL pipeline;
determine two or more ETL resources from the plurality of ETL resources that are best fit to carry out the at least one action based on the at least one characteristic of each of the plurality of ETL resources; and
run each of at least one container from the at least one container image that includes files and parameters for running the at least one container, the at least one container utilizing two or more ETL resources of the plurality of ETL resources, wherein at least one of the two or more ETL resources comprises at least one server that hosts the at least one container, the at least one container isolating user space of the at least one container from other processes while sharing kernel space with the other processes on the at least one server.