Methods, systems, and portal using software containers for accelerating aspects of data analytics application development and deployment
View Patent ↗The present disclosure relates to methods and systems for accelerating the development and distribution of data science workloads, including a consistent, portable and pre-configured data science workspace for development of data science containers allowing for the creation of a standardized, modular and reusable library of data science containers that can be maintained, extended and reused in a clear and repeatable manner. The containers may be submitted to a build and deployment process that ensures consistency across multiple environments in terms of the application code and the operating system environment. Runtime execution may be managed through the authoring of definitions which detail aspects of how the workload should operate within a certain environment.
1. A system for accelerating the development and distribution of data science workloads to an active analytics environment, the system comprising:
a development environment comprising a plurality of computing systems that are isolated from directly deploying jobs to the active analytics environment, each system of the plurality of computing systems comprising
a developer workspace, configured for execution in a virtual environment executable upon a plurality of different operating system platforms such that the developer workspace maintains a same interface and experience regardless of underlying operating system platform of the plurality of different operating system platforms, the developer workspace comprising
a set of basic application dependencies for sharing across multiple development projects created within a plurality of developer workspaces of the development environment, wherein
in combination with application code, the set of basic application dependencies support encapsulated functioning of tasks developed as one or more software containers within the developer workspace;
a version control system in communication with the plurality of developer workspaces, the version control system for
managing sharing of in-progress software projects between an originating developer workspace of the plurality of developer workspaces and at least one additional developer workspace of the plurality of developer workspaces, and
maintaining historic versions of software projects developed within the plurality of developer workspaces; and
a build and deployment process pipeline in communication with the version control system for automatically launching experiments produced in the plurality of developer workspaces and stored in the version control system as jobs executing upon the active analytics environment, the build and deployment process pipeline comprising
a build agent for
receiving a software container from the version control system, and
building a software container image of the software container,
a container repository for staging the software container image in preparation for deployment, and
a software container service for
retrieving the software container image from the container repository, and
automatically deploying the software container image in the active analytics environment;
wherein the active analytics environment hosts a plurality of analytics jobs on behalf of a plurality of clients, and wherein
one or more software processes of the software container image access analytics data from an analytics data store of the active analytics environment while remaining isolated from the plurality of analytics jobs.
2. The system of claim 1 , wherein the version control system comprises at least one base project template including the application dependencies, wherein
the developer workspace retrieves the base project template including the application dependencies from the version control system for developing a new software container.
3. The system of claim 2 , wherein the base project template comprises a service composition tool for enabling container services exposed by the software container.
4. The system of claim 3 , wherein the service composition tool comprises a test environment destruction sequence for destroying an errant test environment.
5. The system of claim 1 , wherein the version control system supports layers of access rights for allowing each additional developer workspace of the plurality of developer workspaces write access to a respective first subset of the plurality of developer workspaces, read-only access to a respective second subset of the plurality of developer workspaces, and no access to a respective third subset of the plurality of developer workspaces.
6. The system of claim 1 , wherein deployment of the software container image is managed in part based upon a task definition, the task definition comprising:
an identifier identifying a software container image within the software repository; and
at least one of memory requirements, CPU requirements, and identification of at least one shared data volume.
7. The system of claim 6 , wherein the task definition comprises at least one additional identifier identifying another software container image designed to interoperate with the software container image.
8. The system of claim 7 , wherein the version control system further maintains a job definition associated with the software container image and the another software container image, wherein the job definition defines scheduling of task execution between the software container image and the another software container image.
9. The system of claim 8 , wherein the job definition defines at least one workflow defining task timing and at least one of task relationships and task dependencies.
10. The system of claim 1 , further comprising:
a build management system for managing builds of a plurality of software containers comprising the software container, the build management system configured to allocate a plurality of build agents including the build agent.
11. The system of claim 10 , wherein build agent allocation is performed in real time based upon a number of and a timing of release of software containers into the build and deployment process pipeline.
12. The system of claim 10 , wherein the build management system comprises a scanning module for scanning contents of the software container to detect a plurality of security vulnerabilities.
13. The system of claim 1 , wherein the build and deployment process pipeline is a cloud-based platform with scalable resource allocation.
14. The system of claim 1 , wherein the plurality of computing systems are virtual machines.
15. A method for accelerating the development and distribution of data science workloads to an active analytics environment, the method comprising:
establishing, within a development environment comprising a plurality of developer computing devices that are isolated from directly deploying jobs to the active analytics environment, a plurality of developer workspaces, each developer workspace
executing in a virtual environment independent of underlying hardware and operating system of a respective developer computing device of the plurality of developer computing devices,
presenting a respective developer interfacing with each computing device of the plurality of developer computing devices with a consistent development interface and tool set for developing software containers for performing data analytics tasks, and
maintaining, at the respective developer workspace, a same interface and experience regardless of the underlying operating system, wherein the respective developer workspace includes
a set of basic application dependencies for sharing across multiple development projects created within a plurality of developer workspaces of the development environment, wherein
in combination with application code, the set of basic application dependencies support encapsulated functioning of tasks developed as one or more software containers within the developer workspace;
populating, from a version control system within the development environment, each developer workspace of the plurality of developer workspaces with a same build template, the build template comprising a plurality of application dependencies;
coordinating, by the version control system, collaboration between the plurality of developer workspaces by enabling sharing of a portion of at least one of respective application code and respective test code between the plurality of developer workspaces;
maintaining, by the version control system, historic versions of software projects developed within the plurality of developer workspaces;
receiving, at the version control system from a first developer workspace of the plurality of developer workspaces, instructions to release a development project comprising application code of the first developer workspace and the test code of the first developer workspace as a software container;
releasing the software container to a build and deployment processing pipeline for building and deployment; and
providing a job definition to a workflow manager of the build and deployment processing pipeline for managing automatic deployment of the software container as a software image into the active analytics environment, wherein
one or more software processes of the software image access analytics data from an analytics data store of the active analytics environment while remaining isolated from a plurality of analytics jobs running on behalf of a plurality of clients in the active analytics environment.
16. The method of claim 15 , wherein the plurality of application dependencies comprises a plurality of standardized development libraries for developing data analytics tasks.
17. The method of claim 15 , wherein the software container comprises a dependent runtime for executing the software image independent of underlying hardware and operating system of the active analytics environment.
18. The method of claim 15 , wherein the build template comprises a template script for configuring the application dependencies, wherein the template script is editable by the developer of a given computing device of the plurality of developer computing devices to alter or extend the plurality of application dependencies.
19. The method of claim 15 , wherein the software container comprises at least one test data set for combining with data accessible in the active analytics environment without releasing the test data set within the active analytics environment.
20. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by processing circuitry, cause the processing circuitry to:
establish, within a development environment comprising a developer computing device that is isolated from directly deploying jobs to an active analytics environment, a developer workspace for
executing in a virtual environment independent of underlying hardware and operating system of the developer computing device, and
presenting a developer interfacing with the computing device with a development interface and tool set for developing software containers for performing data analytics tasks in the active analytics environment,
maintaining, at the developer workspace, a same interface and experience regardless of the underlying operating system, wherein the developer workspace includes
a set of basic application dependencies for sharing across multiple development projects created within a plurality of developer workspaces of the development environment, wherein
in combination with application code, the set of basic application dependencies support encapsulated functioning of tasks developed as one or more software containers within the developer workspace;
populate the developer workspace with a build template, comprising
a plurality of application dependencies,
an application code structure for developing application code,
a test code structure for developing test code for testing the application code, and
a job definition structure for developing a job definition defining timing coordination for tasks performed through execution of the application code;
enable sharing of in-progress software projects between an originating developer workspace of the plurality of developer workspaces;
maintain historic versions of software projects developed within the plurality of developer workspaces;
receive instruction to release a development project comprising the application code and test code as a software container;
release the software container to a build and deployment processing pipeline for building and deployment; and
provide the job definition to a workflow manager of the build and deployment processing pipeline for managing automatic deployment of the software container as a software image in the active analytics environment, wherein
one or more software processes of the software image access analytics data from an analytics data store of the active analytics environment while remaining isolated from a plurality of analytics jobs running on behalf of a plurality of clients in the active analytics environment.