Method and system for processing data
A method for processing data includes receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number, and determining a migration overhead for adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, wherein the states are intermediate results of processing the data and the migration overhead includes a latency and a bandwidth shortage incurred for migrating the states. Based on the determined migration overhead, the states are migrated between the first number of consumer instances and the second number of consumer instances, and thereafter the data is processed based on the second distribution of the states at the second number of consumer instances.
1. A method for processing data, comprising:
receiving an adjustment request for adjusting a number of consumer instances from a first number of consumer instances to a second number of consumer instances;
determining a migration overhead for adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data, the determining being based on a consistent hashing algorithm identifying K/n states to be migrated, wherein K is a total number of states across the first number of consumer instances and n is the second number;
based on the determined migration overhead, migrating the states between the first number of the consumer instances and the second number of the consumer instances; and
thereafter processing the data based on the second distribution of the states at the second number of the consumer instances.
2. The method according to claim 1 , further comprising:
at one consumer instance of the second number of consumer instances before the migrating of the states is completed, caching data dispatched from a producer instance and processing the cached data after a relevant portion of the states are migrated.
3. The method according to claim 2 , wherein the data is dispatched to the second number of consumer instances according to the second distribution of the states.
4. The method according to claim 2 , wherein the relevant portion of the states are migrated to the one consumer instance in response to receipt of a set of adjustment indicators for the adjustment of the states among the consumer instances.
5. The method according to claim 4 , further comprising:
before the set of adjustment indicators are received, processing the data at the one consumer instance based on the first distribution of the states.
6. The method according to claim 1 , further including at least one of:
in response to determining that a workload for processing the data increases, sending the adjustment request for increasing the number of the consumer instances from the first number to the second number; or
in response to determining that a workload for processing the data decreases, sending the adjustment request for decreasing the number of the consumer instances from the first number to the second number.
7. The method according to claim 1 , further comprising:
dynamically adjusting computing resources at runtime of a stream processing application which operates in accordance with a Directed Acyclic Graph (DAG) topology.
8. A system for processing data, comprising:
one or more processors;
a memory coupled to at least one processor of the one or more processors, the memory storing computer program instructions which, when executed by the processors, cause the system to execute a method for processing data, the method comprising:
receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number;
determining a migration overhead for adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data, the determining being based on a consistent hashing algorithm identifying K/n states to be migrated, wherein K is a total number of states across the first number of consumer instances and n is the second number;
based on the determined migration overhead, migrating the states between the first number of the consumer instances and the second number of the consumer instances; and
thereafter processing the data based on the second distribution of the states at the second number of the consumer instances.
9. The system according to claim 8 , further comprising:
at one consumer instance of the second number of consumer instances before the migrating of the states is completed, caching data dispatched from a producer instance and processing the cached data after a relevant portion of the states are migrated.
10. The system according to claim 9 , wherein the data is dispatched to the second number of consumer instances according to the second distribution of the states.
11. The system according to claim 9 , wherein the relevant portion of the states are migrated to the one consumer instance in response to receipt of a set of adjustment indicators for the adjustment of the states among the consumer instances.
12. The system according to claim 11 , wherein the method further includes:
before the set of adjustment indicators are received, processing the data at the one consumer instance based on the first distribution of the states.
13. The system according to claim 8 , wherein the method further includes at least one of:
in response to determining that a workload for processing the data increases, sending the adjustment request for increasing the number of the consumer instances from the first number to the second number; or
in response to determining that a workload for processing the data decreases, sending the adjustment request for decreasing the number of the consumer instances from the first number to the second number.
14. The system according to claim 8 , wherein the method further includes:
dynamically adjusting computing resources at runtime of a stream processing application which operates in accordance with a Directed Acyclic Graph (DAG) topology.
15. A computer program product having a non-transitory computer readable medium which stores a set of instructions which, when carried out by computerized circuitry, cause the computerized circuitry to perform a method of processing data, the method comprising:
receiving an adjustment request for adjusting a number of consumer instances from a first number to a second number;
determining a migration overhead for adjusting a first distribution of states associated with the first number of consumer instances to a second distribution of the states associated with the second number of consumer instances, the states being intermediate results of processing the data, the determining being based on a consistent hashing algorithm identifying K/n states to be migrated, wherein K is a total number of states across the first number of consumer instances and n is the second number;
based on the determined migration overhead, migrating the states between the first number of the consumer instances and the second number of the consumer instances; and
thereafter processing the data based on the second distribution of the states at the second number of the consumer instances.
16. The computer program product according to claim 15 , wherein the method further includes:
at one consumer instance of the second number of consumer instances before the migrating of the states is completed, caching data dispatched from a producer instance and processing the cached data after a relevant portion of the states are migrated.
17. The computer program product according to claim 16 , wherein the data is dispatched to the second number of consumer instances according to the second distribution of the states.