Resource allocation for computer processing
A job that receives as input a data stream is executed. For the job, it is iteratively determined a backlog growth over a first period of time; a backlog volume; and whether to adjust a quantity of processing resources. For each iteration for which the quantity of processing resources allocated to the job are determined to be adjusted, adjusting the quantity of processing resources allocated to the job. For each iteration for which the quantity of processing resources allocated to the job are determined not to be adjusted, maintaining the quantity of processing resources allocated to the job.
1 . A method for allocating resources in a computer system, the method comprising:
executing a job that receives as input a data stream;
determining whether a backlog volume of data that is to be processed by the computer system is greater than, less than, or equal to a predetermined threshold;
determining whether backlog growth of the data that is to be processed is positive, negative, or zero;
based on determining that the backlog volume of data is greater than or equal to the predetermined threshold and determining that the backlog growth is positive-, adjusting a quantity of processing resources allocated to the job, wherein adjusting the quantity of processing resources allocated to the job comprises increasing the quantity of processing resources allocated to the job; and
maintaining the quantity of processing resources allocated to the job when the backlog volume of data that is to be processed by the computer system is below the predetermined threshold and the backlog growth is determined to be positive; and
executing the job using the quantity of processing resources allocated to the job.
2 . The method of claim 1 , wherein the backlog volume of data is a measure of unprocessed data in the received data stream that is to be input into the job.
3 . The method of claim 1 , wherein the backlog volume is measured in terms of a data size.
4 . The method of claim 1 , wherein resources include processor resources or storage resources.
5 . The method of claim 1 , the method comprising:
after the adjusting of the quantity of processing resources:
determining that the quantity of processing resources allocated to the job are to be adjusted to a level of resources used by the computer system prior to the increase in the quantity of processing resources.
6 . A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
executing a job that receives as input a data stream;
determining whether a backlog volume of data that is to be processed by the computer system is greater than, less than, or equal to a predetermined threshold;
determining whether backlog growth of the data that is to be processed is positive, negative, or zero;
based on determining that the backlog volume of data is greater than or equal to the predetermined threshold and determining that the backlog growth is positive, adjusting a quantity of resources allocated to the job, wherein adjusting the quantity of resources allocated to the job comprises increasing the quantity of resources allocated to the job in response to the backlog volume being determined to be greater than or equal to the predetermined threshold;
maintaining the quantity of resources allocated to the job when the backlog volume of data that is to be processed by the computer system is below the predetermined threshold and the backlog growth is determined to be positive; and
executing the job using the quantity of processing resources allocate to the job.
7 . The system of claim 6 , wherein the backlog volume of data is a measure of unprocessed data in the received data stream that is to be input into the job.
8 . The system of claim 6 , wherein the backlog volume is measured in terms of a data size.
9 . The system of claim 6 , wherein resources include processor resources or storage resources.
10 . The system of claim 6 , the operations comprising:
after the adjusting of the quantity of resources:
determining that the quantity of resources allocated to the job are to be adjusted to a level of resources used by the computer system prior to the increase in the quantity of resources.
11 . A non-transitory computer-readable storage device having stored thereon instructions, which, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:
executing a job that receives as input a data stream;
determining whether a backlog volume of data that is to be processed by the computer system is greater than or equal to a predetermined threshold;
determining whether backlog growth of the data that is to be processed is positive, negative, or zero;
based on determining that the backlog volume of data is greater than or equal to the predetermined threshold and determining that the backlog growth is positive, adjusting a quantity of resources allocated to the job, wherein adjusting the quantity of resources allocated to the job comprises increasing the quantity of resources allocated to the job;
maintaining the quantity of resources allocated to the job when the backlog volume of data that is to be processed by the computer system is below the predetermined threshold and the backlog growth is determined to be positive; and
executing the job using the quantity of processing sources allocated to the job.
12 . The computer-readable medium of claim 11 , wherein the backlog volume of data is a measure of unprocessed data in the received data stream that is to be input into the job.
13 . The computer-readable medium of claim 11 , wherein the backlog volume is measured in terms of a data size.
14 . The computer-readable medium of claim 11 , wherein resources include processor resources or storage resources.
15 . The computer-readable medium of claim 11 , the operations comprising:
after the adjusting of the quantity of resources:
determining that the quantity of resources allocated to the job are to be adjusted to a level of resources used by the computer system prior to the increase in the quantity of resources.