Mapping components of a non-distributed environment to a distributed environment
Embodiments of the present invention disclose a method, a computer program product, and a computer system for mapping components of non-distributed environments to distributed environments. A computer receives a data pipeline configured for a non-distributed environment and identifies one or more bottleneck components of the data pipeline. In addition, the computer converts data used in the pipeline to a format compatible with a distributed environment and installs the necessary computing libraries necessary for operating the pipeline within the distributed environment. The computer further converts the code of the pipeline to a code that is compatible with the distributed environment and optimizes components of the pipeline for use in the distributed environment.
1. A method for mapping one or more components of a data pipeline from a non-distributed computing environment to a distributed computing environment, the method comprising:
a computer identifying at least one bottleneck component of one or more components comprising the data pipeline when implemented within the non-distributed computing environment, wherein identifying the at least one bottleneck component is based on a hybrid approach comprising a memory driven approach and a performance driven approach, wherein the memory driven approach uses top command and other memory commands to identify which components are taking a longest time to perform a task, wherein the performance driven approach utilizes system timing commands to identify statistics describing how long various functions take to perform, wherein the hybrid approach sets thresholds through parameters for each component to complete a task and such thresholds include a maximum memory available for a component within the pipeline and a maximum acceptable time for completion of a component within the data pipeline, and further comprises:
increasing an input to the one or more components comprising the data pipeline;
determining that a performance of the at least one component is less than proportional to the increase of the input;
the computer converting data corresponding to the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment;
the computer installing libraries corresponding to an architecture of the distributed computing environment; and
the computer converting code of the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment.
2. The method of claim 1 , further comprising:
the computer measuring a performance of the at least one bottleneck component when implemented within the distributed computing environment; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on the measured performance of the at least one bottleneck.
3. The method of claim 2 , further comprising:
the computer determining whether the at least one bottleneck component is capable of being parallelized; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on determining that the at least one bottleneck component is capable of being parallelized.
4. The method of claim 1 , wherein identifying the at least one bottleneck component further comprises at least one of:
the computer identifying at least one component of the one or more components having a greatest memory usage; and
the computer identifying at least one component of the one or more components having a memory usage exceeding a threshold value.
5. The method of claim 1 , wherein identifying the at least one bottleneck component further comprises at least one of:
the computer identifying at least one component of the one or more components taking a longest time to complete a task; and
the computer identifying at least one component of the one or more components taking longer than a threshold time to complete a task.
6. The method of claim 1 , wherein identifying the at least one bottleneck component further comprises:
the computer generating a model for operating the data pipeline having one or more parameters and one or more constraints, wherein the one or more parameters include memory usage and time;
the computer determining a lowest memory usage and shortest time based on the model; and
the computer identifying at least one component of the one or more components that fails to perform at the lowest memory usage or the shortest time.
7. The method of claim 1 , wherein converting code of the at least one bottleneck further comprises:
the computer determining that code of the at least one bottleneck component is incompatible with code of the non-distributed computing environment; and
the computer generating a wrapper function enabling compatibility between code of the at least one bottleneck component and code of the non-distributed computing environment.
8. A computer program product for mapping one or more components of a data pipeline from a non-distributed computing environment to a distributed computing environment, the computer program product comprising:
one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, the program instructions comprising:
program instructions to identify at least one bottleneck component of one or more components comprising the data pipeline when implemented within the non-distributed computing environment, wherein identifying the at least one bottleneck component is based on a hybrid approach comprising a memory driven approach and a performance driven approach, wherein the memory driven approach uses top command and other memory commands to identify which components are taking a longest time to perform a task, wherein the performance driven approach utilizes system timing commands to identify statistics describing how long various functions take to perform, wherein the hybrid approach sets thresholds through parameters for each component to complete a task and such thresholds include a maximum memory available for a component within the pipeline and a maximum acceptable time for completion of a component within the data pipeline;
program instructions to increase an input to the one or more components comprising the data pipeline;
program instructions to determine that a performance of the at least one component is less than proportional to the increase of the input;
program instructions to convert data corresponding to the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment;
program instructions to install libraries corresponding to an architecture of the distributed computing environment; and
program instructions to convert code of the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment.
9. The computer program product of claim 8 , further comprising:
program instructions to measure a performance of the at least one bottleneck component when implemented within the distributed computing environment; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on the measured performance of the at least one bottleneck.
10. The computer program product of claim 9 , further comprising:
program instructions to determine whether the at least one bottleneck component is capable of being parallelized; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on determining that the at least one bottleneck component is capable of being parallelized.
11. The computer program product of claim 8 , wherein program instructions to identify the at least one bottleneck component further comprise at least one of:
program instructions to identify at least one component of the one or more components having a greatest memory usage; and
program instructions to identify at least one component of the one or more components having a memory usage exceeding a threshold value.
12. The computer program product of claim 8 , wherein program instructions to identify the at least one bottleneck component further comprise at least one of:
program instructions to identify at least one component of the one or more components taking a longest time to complete a task; and
program instructions to identify at least one component of the one or more components taking longer than a threshold time to complete a task.
13. The computer program product of claim 8 , wherein program instructions to identify the at least one bottleneck component further comprise:
program instructions to generate a model for operating the data pipeline having one or more parameters and one or more constraints, wherein the one or more parameters include memory usage and time;
program instructions to determine a lowest memory usage and shortest time based on the model; and
program instructions to identify at least one component of the one or more components that fails to perform at the lowest memory usage or the shortest time.
14. The computer program product of claim 8 , wherein program instructions to convert code of the at least one bottleneck further comprise:
program instructions to determine that code of the at least one bottleneck component is incompatible with code of the non-distributed computing environment; and
program instructions to generate a wrapper function enabling compatibility between code of the at least one bottleneck component and code of the non-distributed computing environment.
15. A computer system for mapping one or more components of a data pipeline from a non-distributed computing environment to a distributed computing environment, the computer system comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising:
program instructions to identify at least one bottleneck component of one or more components comprising the data pipeline when implemented within the non-distributed computing environment, wherein identifying the at least one bottleneck component is based on a hybrid approach comprising a memory driven approach and a performance driven approach, wherein the memory driven approach uses top command and other memory commands to identify which components are taking a longest time to perform a task, wherein the performance driven approach utilizes system timing commands to identify statistics describing how long various functions take to perform, wherein the hybrid approach sets thresholds through parameters for each component to complete a task and such thresholds include a maximum memory available for a component within the pipeline and a maximum acceptable time for completion of a component within the data pipeline;
program instructions to increase an input to the one or more components comprising the data pipeline;
program instructions to determine that a performance of the at least one component is less than proportional to the increase of the input;
program instructions to convert data corresponding to the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment;
program instructions to install libraries corresponding to an architecture of the distributed computing environment; and
program instructions to convert code of the at least one bottleneck component from a format compatible with the non-distributed computing environment to a format compatible with the distributed computing environment.
16. The computer system of claim 15 , further comprising:
program instructions to measure a performance of the at least one bottleneck component when implemented within the distributed computing environment; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on the measured performance of the at least one bottleneck.
17. The computer system of claim 16 , further comprising:
program instructions to determine whether the at least one bottleneck component is capable of being parallelized; and
wherein identifying the at least one bottleneck component of one or more components comprising the data pipeline is further based on determining that the at least one bottleneck component is capable of being parallelized.
18. The computer system of claim 15 , wherein program instructions to identify the at least one bottleneck component further comprise at least one of:
program instructions to identify at least one component of the one or more components having a greatest memory usage; and
program instructions to identify at least one component of the one or more components having a memory usage exceeding a threshold value.
19. The computer system of claim 15 , wherein program instructions to identify the at least one bottleneck component further comprise at least one of:
program instructions to identify at least one component of the one or more components taking a longest time to complete a task; and
program instructions to identify at least one component of the one or more components taking longer than a threshold time to complete a task.
20. The computer system of claim 15 , wherein program instructions to identify the at least one bottleneck component further comprise:
program instructions to generate a model for operating the data pipeline having one or more parameters and one or more constraints, wherein the one or more parameters include memory usage and time;
program instructions to determine a lowest memory usage and shortest time based on the model; and
program instructions to identify at least one component of the one or more components that fails to perform at the lowest memory usage or the shortest time.