Big data application lifecycle management
Aspects of the present disclosure involve systems, methods, devices, and the like for creating an application lifecycle management platform for big data applications. In one embodiment the lifecycle management platform can include a multiple-layer container file that integrates multiple big-data tools/platforms. The system may create a generic template application, create a build environment for the generic template application, create a test environment for the generic template application, and run the built generic template application in the test environment prior to the user writing any new code in the generic template application. In one embodiment, the test environment includes a container management system or virtual machine that launches the big data application (which may be the generic template application before a developer edits the file) on a separate big-data server cluster.
1. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:
receiving an indication to create a big-data application;
in response to receiving the indication, generating an application instance using a big-data container template that comprises a multi-layer container configured to manage a lifecycle of the big-data application, wherein the multi-layer container comprises:
a first container layer comprising operating system settings and security settings;
a second container layer comprising big-data settings and libraries common to a plurality of big-data applications;
a third container layer comprising user settings and a test application; and
a fourth container layer comprising a big-data application source code and big-data application specific settings; and
configuring a build environment and a test environment for the application instance.
2. The non-transitory machine-readable medium of claim 1 , wherein the big-data container template comprises a DOCKER image.
3. The non-transitory machine-readable medium of claim 1 , wherein the first container layer further comprises at least one of an operating system, a set of platform tools, or a set of platform security tools, and wherein the second container layer further comprises an orchestration monitoring client and a big-data file system client.
4. The non-transitory machine-readable medium of claim 1 , wherein the operations further comprise:
receiving a trigger indicating that the big-data application source code has been updated; and
in response to receiving the trigger generating a second application instance based on the updated big-data application source code.
5. The non-transitory machine-readable medium of claim 4 , wherein the operations further comprise re-testing the second application instance in the test environment.
6. The non-transitory machine-readable medium of claim 1 , wherein the first container layer further comprises a customized operating system configured with one or more specialized developer tools.
7. The non-transitory machine-readable medium of claim 1 , wherein the operations further comprise testing the application instance in the test environment.
8. A system comprising:
a non-transitory memory; and
one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
receiving an indication to create a big-data application;
generating an application instance using a big-data container template that comprises a multi-layer container configured to manage a lifecycle of the big-data application, wherein the multi-layer container comprises:
a first container layer comprising operating system settings and security settings;
a second container layer comprising big-data settings and libraries common to a plurality of big-data applications;
a third container layer comprising user settings and a test application; and
a fourth container layer comprising a big-data application source code and big-data application specific settings; and
configuring a build environment and a test environment for the application instance.
9. The system of claim 8 , wherein the operations further comprise testing the application instance in the test environment.
10. The system of claim 8 , wherein the operations further comprise:
receiving a trigger indicating that the big-data application source code has been updated; and
in response to receiving the trigger, generating a second application instance based on the updated big-data application source code.
11. The system of claim 10 , wherein the operations further comprise re-testing the second application instance in the test environment.
12. The system of claim 8 , wherein the operations further comprise:
configuring the application instance to process operational data; and
deploying the application instance in a production environment.
13. The system of claim 12 , wherein the deploying comprises performing a staggering deployment of the application instance into a plurality of data clusters.
14. The system of claim 8 , wherein the operations further comprise executing the application instance on a production server cluster according to a predetermined schedule.
15. The system of claim 14 , wherein the operations further comprise determining the schedule based on data dependencies between two or more big-data applications.
16. A method, comprising:
receiving, by one or more hardware processors, an indication to create a big-data application;
generating, by the one or more hardware processors, an application instance using a big-data container template that comprises a multi-layer container configured to manage a lifecycle of the big-data application, wherein the multi-layer container comprises:
a first container layer comprising operating system settings and security settings;
a second container layer comprising big-data settings and libraries common to a plurality of big-data applications;
a third container layer comprising user settings and a test application; and
a fourth container layer comprising a big-data application source code and big-data application specific settings; and
configuring, by the one or more hardware processors, a build environment and a test environment for the application instance.
17. The method of claim 16 , further comprising executing the application instance in a production environment.
18. The method of claim 17 , further comprising:
collecting resource and user usage statistics based on the executing the application instance in the production environment during a first time period; and
determining a schedule for executing the application instance during a second time period based on the collected resource and user usage statistics.
19. The method of claim 16 , wherein the test environment comprises a device running a virtual machine.
20. The method of claim 19 , further comprising causing the virtual machine to launch multiple versions of the application instance.