Distributed processing system throttling using a timestamp
Methods, systems, and apparatus are described for throttling a distributed processing system. In one aspect, a method includes identifying records being processed by a distributed processing system that performs agent processes, each of the records including a corresponding timestamp; determining, based on timestamps of the records that have been processed by a first agent process, a first agent progress; identifying a dependent agent process performed by the distributed processing system, wherein the dependent agent process processes only records that have been processed by the first agent process; determining, based on timestamps of records that have been processed by the dependent agent process, a dependent agent progress; and throttling performance of the first process based on the first agent progress and the dependent agent progress.
1. A method for data processing a set of records by a distributed processing system, the method comprising:
determining, by one or more processors, a first agent progress in processing the set of records, wherein the first agent progress is based on a first time difference between a first timestamp of an oldest record of the set of records processed by a first agent process and entered into a queue for a second agent process and a current system time, wherein the second agent process is a dependent agent process of the first agent process and processes records that have first been processed by the first agent process;
determining, by the one or more processors, a second agent progress in processing the set of records, wherein the second agent progress is based on a second time difference between a second timestamp of an oldest record of the set of records processed by the second agent process and the current system time, wherein the first timestamp of the oldest record of the set of records processed by the first agent process and the second timestamp of the oldest record of the set of records processed by the second agent process each correspond a time the respective oldest record was received by the distributed processing system;
determining, by the one or more processors, a difference between the first agent progress and the second agent progress; and
throttling, by the one or more processors, performance of the first agent process, based on the difference between the first agent progress and the second agent progress meeting a threshold measure.
2. The method of claim 1 , wherein each record of the set of records processed by the first agent process is provided to the queue.
3. The method of claim 1 , wherein throttling performance of the first agent process includes reducing the processing speed of data processors performing the first agent process.
4. The method of claim 1 , wherein throttling performance of the first agent process includes removing one or more data processors assigned to perform the first agent process.
5. The method of claim 4 , further comprising:
determining, by the one or more processors, based in part on the first agent progress, to increase a second agent progress for the second agent process by increasing a number of data processors assigned to perform the second agent process.
6. The method of claim 5 , wherein the data processors assigned to perform the second agent process are the one or more removed data processors.
7. The method of claim 1 , wherein the records being processed are query records.
8. A system comprising:
a data processing apparatus; and
a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising:
determining a first agent progress in processing a set of records, wherein the first agent progress is based on a first timestamp of an oldest record of the set of records processed by a first agent process and entered into a queue for a second agent process, and a current system time, wherein the second agent process is a dependent agent process of the first agent process and processes records that have first been processed by the first agent process;
determining a second agent progress in processing the set of records, wherein the second agent progress is based on a second time difference between a second timestamp of an oldest record of the set of records processed by the second agent process and the current system time, wherein the first timestamp of the oldest record of the set of records processed by the first agent process and the second timestamp of the oldest record of the set of records processed by the second agent process each correspond to a time the respective record was received by the processing apparatus;
determining a difference between the first agent progress and the second agent progress; and
throttling performance of the first agent process based on the difference between the first agent progress and the second agent progress meeting a threshold measure.
9. The system of claim 8 , wherein each record of the set of records processed by the first agent process is provided to the queue.
10. The system of claim 8 , wherein throttling performance of the first agent process includes reducing the processing speed of data processors performing the first agent process.
11. The system of claim 8 , wherein throttling performance of the first agent process includes removing one or more data processors assigned to perform the first agent process.
12. The system of claim 11 , further comprising:
determining, by the one or more processors, based in part on the first agent progress, to increase a second agent progress for the second agent process by increasing a number of data processors assigned to perform the second agent process.
13. The system of claim 12 , wherein the data processors assigned to perform the second agent process are the one or more removed data processors.
14. The system of claim 8 , wherein the records being processed are query records.
15. A non-transitory computer readable medium storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:
determining a first agent progress in processing a set of records, wherein the first agent progress is based on a first time difference between a first timestamp of an oldest record of the set of records processed by a first agent process and entered into a queue for a second agent process and a current system time, wherein the second agent process is a dependent agent process of the first agent process and processes records that have first been processed by the first agent process;
determining a second agent progress in processing the set of records, wherein the second agent progress is based on a second time difference between a second timestamp of an oldest record of the set of records processed by the second agent process and the current system time, wherein the first timestamp of the oldest record of the set of records processed by the first agent process and the second timestamp of the oldest record of the set of records processed by the second agent process each correspond a time the respective oldest record was received by the data processing apparatus;
determining, by the one or more processors, a difference between the first agent progress and the second agent progress; and
throttling performance of the first agent process based on the difference between the first agent progress and the second agent progress meeting a threshold measure.
16. The non-transitory computer readable medium of claim 15 , wherein each record of the set of records processed by the first agent process is provided to the queue.
17. The non-transitory computer readable medium of claim 15 , wherein throttling performance of the first agent process includes reducing the processing speed of data processors performing the first agent process.
18. The non-transitory computer readable medium of claim 15 , wherein throttling performance of the first agent process includes removing one or more data processors assigned to perform the first agent process.
19. The non-transitory computer readable medium of claim 18 , further comprising:
determining, by the one or more processors, based in part on the first agent progress, to increase a second agent progress for the second agent process by increasing a number of data processors assigned to perform the second agent process.
20. The non-transitory computer readable medium of claim 19 , wherein the data processors assigned to perform the second agent process are the one or more removed data processors.