Multi-Tenant Persistent Job History Service for Data Processing Centers

1 . A system for providing access to logs and/or history information for jobs that were processed or run on a cluster that was automatically terminated, the system comprising:

a persistence component, configured to save job history, configuration, and/or log files related to a cluster even after the cluster is terminated;

a terminated job history server, configured to serve requests for logs and histories associated with jobs that ran on terminated clusters; and

a cluster proxy, providing a proxy layer to redirect requests regarding terminated cluster job history, configuration, and/or log files to the terminated job history server.

2 . The system of claim 1 , wherein the persistence component confirms or ensures that the job history, configuration, and/or log files are saved and accessible after cluster termination.

3 . The system of claim 1 , wherein the terminated job history server is implemented once for each type of job.

4 . The system of claim 1 , wherein the terminated job history server is persistent and multi-tenant.

5 . The system of claim 1 , wherein the cluster proxy further maintains a job history server for graphical user interface (GUI) access to jobs it has run.

6 . The system of claim 1 , wherein the cluster proxy is configured to perform authentication, authorization, and/or routing.

7 . The system of claim 6 , wherein the authentication performed by the cluster proxy is based at least in part on cookies verifying that the request is made by an authorized user.

8 . The system of claim 6 , wherein the authorization is performed by the cluster proxy based at least in part on matching hostname information associated with the request with stored node information.

9 . The system of claim 1 , wherein the cluster proxy redirects requests by appending information about a storage facility location and credentials necessary to retrieve the history and/or log files from the terminated job history server.

10 . A method of providing access to logs and/or history information for jobs that were processed or run on a cluster that was automatically terminated, the method comprising:

saving job history and/or log files associated with ephemeral clusters in a persistent storage facility;

receiving a request from a user for information pertaining to a job, the request received at a cluster proxy;

determining by the cluster proxy if the request pertains to a job that was processed or run on a terminated cluster;

upon a determination that the request pertains to a job that was processed or run on a terminated cluster, directing by the cluster proxy the user request to a terminated job history server; and

providing, by the terminated job history server through access to a storage facility, access to logs and/or history information requested by the user.

11 . The method of claim 10 , wherein the cluster proxy appends information about a storage facility and credentials to retrieve the job history and/or log files from storage facility by the terminated job history server.

12 . The method of claim 10 , further comprising:

upon a determination that the request pertains to a job that is currently running on an active cluster, directing by the cluster proxy the user request to the active cluster.

13 . The method of claim 12 , wherein the directing by the cluster proxy the user request to the active cluster comprises directing the request to the relevant job history server of the active cluster.

14 . The method of claim 10 , wherein the request from the user is received via a web server, and wherein the web server sends the request to the cluster proxy.

15 . The method of claim 10 , wherein the cluster proxy is configured to parse any links contained in persistent job history servers to confirm that such links are in useable format and reference data stored at the storage facility.

16 . The method of claim 15 , wherein the storage facility comprises cloud storage.