Distributed job manager for stateful microservices
Two servers implemented as containerized applications may manage the storage of data and the access of that data by compute jobs in a distributed system. A metadata server may distribute data on ingress and assign files to particular storage volumes. The metadata server may then provide a lookup function for files and be configured to distribute a file to other volumes when necessary. A job server may launch jobs as containerized applications and coordinate data access across jobs.
1. A computing system comprising:
a plurality of computing nodes that each include a respective processor, a respective memory module, and a respective communications interface, each computing node being configured to execute a compute job upon request, each computing node including a storage interface configured to communicate with a respective virtual storage volume;
a metadata server, implemented as a containerized application, configured to distribute a plurality of files among the respective virtual storage volumes and to identify upon request the virtual storage volume associated with a designated one of the files; and
a job server, implemented as a containerized application, configured to initiate a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated, wherein the job server is configured to instruct the metadata server to replicate a designated result set associated with a designated one of the jobs across the virtual storage volumes.
2. The computing system recited in claim 1 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.
3. The computing system recited in claim 2 , wherein the one or more containerized applications includes a storage driver configured to manage the respective virtual storage volume.
4. The computing system recited in claim 2 , wherein the metadata server is implemented as a designated containerized application on a designated one of the plurality of computing nodes.
5. The computing system recited in claim 2 , wherein the job server is implemented as a designated containerized application on a designated one of the plurality of computing nodes.
6. The computing system recited in claim 2 , wherein each of the compute jobs is implemented as one of the containerized applications.
7. The computing system recited in claim 1 , wherein the job server is further configured to monitor each of the compute jobs for completion.
8. The computing system recited in claim 7 , wherein the job server instructs the metadata server to replicate a designated result set when it is determined that the designated compute job has completed.
9. The computing system recited in claim 1 , wherein the identification of the virtual storage volume associated with a designated file includes applying a consistent hashing function to data or metadata associated with the designated file.
10. The computing system recited in claim 1 , wherein each of the jobs includes one or more tasks associated with training a machine learning model using data stored in the files stored in the virtual storage volume associated with the compute node on which the job is instantiated.
11. A method comprising:
distributing via a metadata serve, implemented as a containerized application, a plurality of files among a plurality of virtual storage volumes, each of the virtual storage volumes in communication with a respective computing node via a respective storage interface, each of the computing nodes including a respective processor, a respective memory module, and a respective communications interface, wherein the metadata server is configured to identify upon request the virtual storage volume associated with a designated one of the files; and
initiating via a job server, implemented as a containerized application, a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated, wherein the job server is configured to instruct the metadata server to replicate a designated result set associated with a designated one of the jobs across the virtual storage volumes.
12. The method recited in claim 11 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.
13. The method recited in claim 12 , wherein the one or more containerized applications includes a storage driver configured to manage the respective virtual storage volume.
14. The method recited in claim 12 , wherein each of the compute jobs is associated with wherein each of the compute jobs is implemented as one of the containerized applications.
15. The method recited in claim 12 , wherein the job server is further configured to monitor each of the compute jobs for completion.
16. The method recited in claim 15 , wherein the job server instructs the metadata server to replicate a designated result set when it is determined that the designated compute job has completed.
17. The method recited in claim 11 , wherein the identification of the virtual storage volume associated with a designated file includes applying a consistent hashing function to data or metadata associated with the designated file.
18. One or more non-transitory machine-readable media having instructions stored thereon for performing a method, the method comprising:
distributing via a metadata serve, implemented as a containerized application, a plurality of files among a plurality of virtual storage volumes, each of the virtual storage volumes in communication with a respective computing node via a respective storage interface, each of the computing nodes including a respective processor, a respective memory module, and a respective communications interface, wherein the metadata server is configured to identify upon request the virtual storage volume associated with a designated one of the files; and
initiating via a job server, implemented as a containerized application, a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated, wherein the job server is configured to instruct the metadata server to replicate a designated result set associated with a designated one of the jobs across the virtual storage volumes.
19. The one or more non-transitory machine-readable media recited in claim 18 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.
20. The one or more non-transitory machine-readable media recited in claim 18 , wherein the job server is further configured to monitor each of the compute jobs for completion.