Method and system for processing data
Embodiments of the present invention relates to a method and system for processing data. Specifically, there is provided a method for processing data, comprising: in response to receiving an adjustment request for adjusting the number of consumer instances from a first number to a second number, determining an adjustment policy on adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data; migrating the states between the first number of the consumer instances and the second number of the consumer instances according to the adjustment policy; and processing the data based on the second distribution of the states at the second number of the consumer instances. In other embodiments, there are further provided a device and system for processing data.
1. A method for processing data, comprising:
receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number;
determining an adjustment policy on adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data;
migrating the states between the first number of the consumer instances and the second number of the consumer instances according to the adjustment policy; and
processing the data based on the second distribution of the states at the second number of the consumer instances; and
wherein the method further comprises:
before migrating the states, at a third number of producer instances that is at upstream of the first number of consumer instances, sending an adjustment indicator to the first number of the consumer instances, the adjusting indicator indicating that data dispatched after the adjustment indicator is dispatched according to the second distributions of the states.
2. The method according to claim 1 , wherein the determining the adjustment policy comprises:
determining migration overhead for adjusting the first distribution of the states to the second distributions of the states; and
determining the adjustment policy based on the migration overhead.
3. The method according to claim 2 , wherein the determining the migration overhead comprises: determining the migration overhead based on a consistent hashing algorithm.
4. The method according to claim 1 , further comprising: after sending the adjustment indicator,
dispatching the data from the third number of producer instances to the second number of consumer instances according to the second distribution of the states.
5. The method according to claim 1 , wherein the migrating the states according to the adjustment policy comprises:
in response to receiving adjustment indicators from all upstream producer instances at a consumer instance among the first number of the consumer instances, migrating a relevant portion of the states to an associated further consumer instance among the second number of the consumer instances according to the adjustment policy.
6. The method according to claim 5 , further comprising:
before receiving adjustment indicators from all upstream producer instances at the consumer instance, processing the data at the consumer instance based on the first distribution of the states.
7. The method according to claim 6 , further comprising:
before receiving adjustment indicators from all upstream producer instances at the consumer instance, caching at the further consumer instance data that is dispatched from the third number of the producer instances.
8. The method according to claim 7 , further comprising:
in response to receiving at the further consumer instance the portion of the states that are migrated from the consumer instance, processing the cached data based on the portion.
9. The method according to claim 1 , further comprising any of:
in response to determining that a workload for processing the data increases, sending the adjustment request for increasing the number of the consumer instances from the first number to the second number; and
in response to determining that a workload for processing the data decreases, sending the adjustment request for decreasing the number of the consumer instances from the first number to the second number.
10. A system for processing data, comprising:
one or more processors;
a memory coupled to at least one processor of the one or more processors;
computer program instructions stored in the memory which, when executed by the at least one processor, cause the system to execute a method for processing data, the method comprising:
receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number;
determining an adjustment policy on adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data;
migrating the states between the first number of the consumer instances and the second number of the consumer instances according to the adjustment policy; and
processing the data based on the second distribution of the states at the second number of the consumer instances; and
wherein the method further comprises:
before migrating the states, at a third number of producer instances that is at upstream of the first number of consumer instances, sending an adjustment indicator to the first number of the consumer instances, the adjusting indicator indicating that data dispatched after the adjustment indicator is dispatched according to the second distributions of the states.
11. The system according to claim 10 , wherein the determining the adjustment policy comprises:
determining migration overhead for adjusting the first distribution of the states to the second distributions of the states; and
determining the adjustment policy based on the migration overhead.
12. The system according to claim 11 , wherein the determining the migration overhead comprises: determining the migration overhead based on a consistent hashing algorithm.
13. The system according to claim 10 , wherein the method further comprises: after sending the adjustment indicator,
dispatching the data from the third number of producer instances to the second number of consumer instances according to the second distribution of the states.
14. The system according to claim 10 , wherein the migrating the states according to the adjustment policy comprises:
receiving adjustment indicators from all upstream producer instances at a consumer instance among the first number of the consumer instances; and
migrating a relevant portion of the states to an associated further consumer instance among the second number of the consumer instances according to the adjustment policy.
15. The system according to claim 14 , wherein the method further comprises:
before receiving adjustment indicators from all upstream producer instances at the consumer instance, processing the data at the consumer instance based on the first distribution of the states.
16. The system according to claim 15 , wherein the method further comprises:
before receiving adjustment indicators from all upstream producer instances at the consumer instance, caching at the further consumer instance data that is dispatched from the third number of the producer instances.
17. The system according to claim 16 , wherein the method further comprises:
in response to receiving at the further consumer instance the portion of the states that are migrated from the consumer instance, processing the cached data based on the portion.
18. The system according to claim 10 , wherein the method further comprises:
in response to determining that a workload for processing the data increases, sending the adjustment request for increasing the number of the consumer instances from the first number to the second number; and
in response to determining that a workload for processing the data decreases, sending the adjustment request for decreasing the number of the consumer instances from the first number to the second number.
19. A computer program product having a non-transitory computer readable medium which stores a set of instructions to process data; the set of instructions, when carried out by computerized circuitry, causing the computerized circuitry to perform a method of:
receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number;
determining an adjustment policy on adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data;
migrating the states between the first number of the consumer instances and the second number of the consumer instances according to the adjustment policy; and
processing the data based on the second distribution of the states at the second number of the consumer instances; and
wherein the method further comprises:
before migrating the states, at a third number of producer instances that is at upstream of the first number of consumer instances, sending an adjustment indicator to the first number of the consumer instances, the adjusting indicator indicating that data dispatched after the adjustment indicator is dispatched according to the second distributions of the states.
20. The computer program product as in claim 19 , wherein the computerized circuitry is constructed and arranged to dynamically adjust computing resources at runtime of a stream processing application which operates in accordance with a Directed Acyclic Graph (DAG) topology.