AUTOMATIC RESPONSE TO INEFFICIENT JOBS IN DATA PROCESSING CLUSTERS

1 . A computer-implemented method, comprising:

for each of a plurality of jobs executed by a computing system component, wherein each job comprises an execution of a corresponding job definition:

retrieving metadata about the job from the computing system component; and

calculating an inefficiency metric for the job based on the metadata, wherein a higher inefficiency metric corresponds to a more inefficient job;

ranking the plurality of jobs based on each job's inefficiency metric and selecting one or more top-ranked jobs from the ranking;

selecting one or more job definitions corresponding to the one or more top-ranked jobs; and

sending optimization requests to users associated with the selected job definitions.

2 . The computer-implemented method of claim 1 , wherein:

the computing system component is a data processing cluster that executes logic to:

receive jobs submitted by users; and

for each submitted job:

execute one or more associated tasks to complete the job; and

store metadata about the job; and

the data processing cluster comprises:

multiple data nodes that execute the tasks associated with the submitted jobs;

a first node managing a namespace encompassing the multiple data nodes;

a second node scheduling the tasks to data nodes; and

a third node for storing the job metadata.

3 . The computer-implemented method of claim 1 , wherein sending optimization requests to users associated with the selected job definitions comprises:

for each of the selected job definitions:

if a ticket exists for the job definition, updating the ticket at an issue tracking server; and

if a ticket does not exist for the job definition, opening a ticket for the job definition at the issue tracking server; and

wherein a ticket for a job definition comprises metadata about at least one job that executed the job definition during the time period.

4 . The computer-implemented method of claim 1 , wherein sending optimization requests to users associated with the selected job definitions comprises:

for each user associated with at least one of the selected job definitions, opening a single ticket for the user, wherein the single ticket references all job definitions associated with the user.

5 . The computer-implemented method of claim 1 , wherein calculating the inefficiency metric for a given job based on the metadata comprises:

obtaining one or more factors about the given job from the metadata;

normalizing each of the one or more factors to share a same scale; and

aggregating the one or more factors to yield the inefficiency metric.

6 . The computer-implemented method of claim 5 , wherein the one or more factors comprise at least one of:

a measure of resources allocated to the given job;

a measure of how efficiently the given job used the allocated resources;

a frequency with which the given job was executed during the time period; and

for each other job aside from the given job that executed the job definition during the time period, a measure of how efficiently the other job used the resources that were allocated by the other job.

7 . The computer-implemented method of claim 6 , wherein the allocated resources comprise at least one of:

an amount of memory allocated to the job; and

an amount of central processing unit (CPU) processing allocated to the job.

8 . The computer-implemented method of claim 7 , wherein the measure of how efficiently the given job used the allocated resources is determined by:

calculating a ratio between the amount of memory allocated by the job and a maximum amount of memory used by the job at any one time; or

calculating a ratio between the amount of memory allocated by the job and an average amount of memory used by the job over the duration of the job.

9 . The computer-implemented method of claim 1 , wherein the metadata comprises at least one of:

a number of mapper tasks associated with the job;

a number of reducer tasks associated with the job;

an amount of memory allocated by each of the mapper tasks and reducer tasks associated with the job;

a maximum amount of memory used by each of the mapper tasks and reducer tasks associated with the job; and

an average amount of memory used by each of the mapper tasks and reducer tasks associated with the job.

10 . The computer-implemented method of claim 1 , wherein the method further comprises for at least one of the selected job definitions:

identifying a specific inefficiency in the job definition using metadata associated with one or more jobs corresponding to the job definition; and

modifying the job definition to alleviate the specific inefficiency.

11 . The computer-implemented method of claim 10 , wherein:

the specific inefficiency comprises each of the one or more jobs associated with the job definition being allocated more memory than a maximum amount of memory used by any of the one or more jobs corresponding to the job definition; and

the modification comprises modifying a configuration associated with the job definition to specify a smaller amount of memory to be allocated.

12 . An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

for each of a plurality of jobs executed by a computing system component, wherein each job comprises an execution of a corresponding job definition:

retrieve metadata about the job from the computing system component;

calculate an inefficiency metric for the job based on the metadata, wherein a higher inefficiency metric corresponds to a more inefficient job;

rank the plurality of jobs based on each job's inefficiency metric and select one or more top-ranked jobs from the ranking;

select one or more job definitions corresponding to the one or more top-ranked jobs; and

send optimization requests to users associated with the selected job definitions.

13 . The apparatus of claim 12 , wherein:

the computing system component is a data processing cluster that executes logic to:

receive jobs submitted by users; and

for each submitted job:

execute one or more associated tasks to complete the job; and

store metadata about the job; and

the data processing cluster comprises:

multiple data nodes that execute the tasks associated with the submitted jobs;

a first node managing a namespace encompassing the multiple data nodes;

a second node scheduling the tasks to data nodes; and

a third node for storing the job metadata.

14 . The apparatus of claim 12 , wherein sending optimization requests to users associated with the selected job definitions comprises:

for each of the selected job definitions:

if a ticket exists for the job definition, updating the ticket at an issue tracking server; and

if a ticket does not exist for the job definition, opening a ticket for the job definition at the issue tracking server; and

wherein a ticket for a job definition comprises metadata about at least one job that executed the job definition during the time period.

15 . The apparatus of claim 12 , wherein sending optimization requests to users associated with the selected job definitions comprises:

for each user associated with at least one of the selected job definitions, opening a single ticket for the user, wherein the single ticket references all job definitions associated with the user.

16 . The apparatus of claim 12 , wherein calculating the inefficiency metric for a given job based on the metadata comprises:

obtaining one or more factors about the given job from the metadata;

normalizing each of the one or more factors to share a same scale; and

aggregating the one or more factors to yield the inefficiency metric.

17 . The apparatus of claim 16 , wherein the one or more factors comprise at least one of:

a measure of resources allocated to the given job;

a measure of how efficiently the given job used the allocated resources;

a frequency with which the given job was executed during the time period; and

for each other job aside from the given job that executed the job definition during the time period, a measure of how efficiently the other job used the resources that were allocated by the other job.

18 . The apparatus of claim 17 , wherein the allocated resources comprise at least one of:

an amount of memory allocated to the job; and

an amount of central processing unit (CPU) processing allocated to the job.

19 . The apparatus of claim 18 , wherein the measure of how efficiently the given job used the allocated resources is determined by:

calculating a ratio between the amount of memory allocated by the job and a maximum amount of memory used by the job at any one time; or

calculating a ratio between the amount of memory allocated by the job and an average amount of memory used by the job over the duration of the job.

20 . One or more non-transitory computer-readable storage media storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

for each of a plurality of jobs executed by a computing system component, wherein each job comprises an execution of a corresponding job definition:

retrieving metadata about the job from the computing system component;

calculating an inefficiency metric for the job based on the metadata, wherein a higher inefficiency metric corresponds to a more inefficient job;

ranking the plurality of jobs based on each job's inefficiency metric and selecting one or more top-ranked jobs from the ranking;

selecting one or more job definitions corresponding to the one or more top-ranked jobs; and

sending optimization requests to users associated with the selected job definitions.