Distributed job manager for stateful microservices

1. A computing system comprising:

a plurality of computing nodes that each include a processor and a memory and that are each configured to execute a compute job upon request, each computing node including a storage interface configured to communicate with a respective virtual storage volume;

a metadata server, implemented as a containerized application, configured to distribute a plurality of files among the respective virtual storage volumes; and

a job server, implemented as a containerized application, configured to

initiate a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated; and

instruct the metadata server to replicate a designated result set associated with a designated compute job included in the one or more compute jobs based on a determination that the designated compute job has completed.

2. The computing system recited in claim 1 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.

3. The computing system recited in claim 2 , wherein the one or more containerized applications includes a storage driver configured to manage the respective virtual storage volume.

4. The computing system recited in claim 2 , wherein the metadata server is implemented as a designated containerized application on a designated one of the plurality of computing nodes.

5. The computing system recited in claim 2 , wherein the job server is implemented as a designated containerized application on a designated one of the plurality of computing nodes.

6. The computing system recited in claim 2 , wherein each of the compute jobs is implemented as one of the containerized applications.

7. The computing system recited in claim 1 , wherein the job server is further configured to monitor each of the compute jobs for completion.

8. The computing system recited in claim 1 , wherein the identification of the virtual storage volume associated with a designated file includes applying a consistent hashing function to data or metadata associated with the designated file.

9. The computing system recited in claim 1 , wherein each of the jobs includes one or more tasks associated with training a machine learning model using data stored in the files stored in the virtual storage volume associated with the compute node on which the job is instantiated.

10. A method comprising:

distributing via a metadata server, implemented as a containerized application, a plurality of files among a plurality of virtual storage volumes, each of the virtual storage volumes in communication with a respective computing node via a respective storage interface;

initiating via a job server, implemented as a containerized application, a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated; and

instructing, via the job server, the metadata server to replicate a designated result set associated with a designated compute job included in the one or more compute jobs based on a determination that the designated compute job has completed.

11. The method recited in claim 10 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.

12. The method recited in claim 11 , wherein the one or more containerized applications includes a storage driver configured to manage the respective virtual storage volume.

13. The method recited in claim 11 , wherein each of the compute jobs is implemented as one of the containerized applications.

14. The method recited in claim 11 , wherein the job server is further configured to monitor each of the compute jobs for completion.

15. The method recited in claim 10 , wherein the identification of the virtual storage volume associated with a designated file includes applying a consistent hashing function to data or metadata associated with the designated file.

16. One or more non-transitory machine-readable media having instructions stored thereon for performing a method, the method comprising:

distributing via a metadata server, implemented as a containerized application, a plurality of files among a plurality of virtual storage volumes, each of the virtual storage volumes in communication with a respective computing node via a respective storage interface; and

initiating via a job server, implemented as a containerized application, a respective one or more compute jobs on each of the plurality of computing nodes, each compute job accessing one or more files stored on the respective virtual storage volume associated with the respective computing node on which the respective compute job is initiated; and

instructing, via the job server, the metadata server to replicate a designated result set associated with a designated compute job included in the one or more compute jobs based on a determination that the designated compute job has completed.

17. The one or more non-transitory machine-readable media recited in claim 16 , wherein each of the computing nodes includes a respective container engine application executed by an operating system, the container engine application providing a standardized platform for the instantiation and execution of containerized applications.

18. The one or more non-transitory machine-readable media recited in claim 16 , wherein the job server is further configured to monitor each of the compute jobs for completion.