Dynamic allocation of partitions in a distributed data processing system
A claim is granted over a partition to a consumer. The claim is valid for a period and allows only that consumer to consume data of the partition during the period. The consumer consumes the data until the claim expires or the data in the partition is fully consumed. If the consumer fully consumes the data of the partition prior to the claim expiring, another claim can be granted over another partition to the consumer. Once the claim over the partition expires, the partition is available to be claimed by another consumer.
1 . A method for processing data in a distributed system, the method comprising:
granting a first claim over a first partition of a plurality of partitions to a first consumer of a plurality of consumers, wherein each of the plurality of partitions stores a different portion of the data, wherein each of the plurality of partitions are part of a same data stream group, wherein the first claim is valid for a first period, and wherein the first claim allows only the first consumer out of the plurality of consumers to consume data of the first partition during the first period;
consuming, by the first consumer during the first period, the data in the first partition until the first claim expires or the data in the first partition is fully consumed by the first consumer, wherein the first consumer fully consumes the data of the first partition prior to the first claim expiring, and wherein the first claim is not released until the first claim expires;
responsive to the first consumer fully consuming the data of the first partition prior to the first claim expiring, granting a second claim over a second partition of the plurality of partitions to the first consumer without releasing the first claim, wherein the second claim is valid for a second period, and wherein the second claim allows only the first consumer out of the plurality of consumers to consume data of the second partition during the second period;
consuming, by the first consumer during the second period, the data in the second partition until the second claim expires or the data in the second partition is fully consumed by the first consumer;
granting a third claim over the first partition to a second consumer of the plurality of consumers after the first claim expires, wherein the third claim is valid for a third period, and wherein the third claim allows only the second consumer out of the plurality of consumers to consume data of the first partition during the third period; and
consuming, by the second consumer during the third period, the data in the first partition until the third claim expires or the data in the first partition is fully consumed by the second consumer.
2 . The method of claim 1 , wherein granting the first claim over the first partition includes:
transmitting a partition claim request that asks for an available partition of the data stream group to consume; and
receiving a partition claim response that indicates a grant of the first claim over the first partition to the first consumer.
3 . The method of claim 2 , wherein the first claim is given over any available partition of the data stream group.
4 . The method of claim 1 , wherein the first period, the second period, and the third period are each for a predefined length of time.
5 . The method of claim 1 , wherein the first period, the second period, and the third period each are defined by an expiration time.
6 . The method of claim 1 , wherein consuming the data in the first partition includes reading the data and sending the data to a database in parallel with the reading.
7 . The method of claim 6 , wherein the first consumer waits until all the read data has been sent to the database prior to making a request for a claim over another available partition of the data stream group to continue.
8 . A system, comprising:
a first plurality of servers that each include a partition of a plurality of partitions that are part of a same data stream group;
a second plurality of servers that execute a plurality of consumer applications, wherein each consumer application is capable of performing operations including:
transmitting a first partition claim request that asks for a first available partition of the plurality of partitions to consume, wherein the first partition claim request indicates an identifier of the data stream group;
receiving a first partition claim response that indicates a first grant of a first claim over a first partition of the plurality of partitions, wherein the first claim is valid for a first period, and wherein the first claim allows this consumer to consume data of the first partition during the first period;
consuming, during the first period, data in the first partition until the first claim expires or the data in the first partition is fully consumed;
responsive to fully consuming the data of the first partition prior to the first claim expiring, transmitting a second partition claim request that asks for a second available partition of the plurality of partitions, wherein after the first claim expires the first partition is available to be claimed by any of the plurality of consumer applications;
receiving a second partition claim response that indicates a second grant of a second claim over a second partition of the plurality of partitions, wherein the second claim is valid for a second period, and wherein the second claim allows this consumer to consume data of the second partition during the second period; and
consuming, during the second period, data in the second partition until the second claim expires or the data in the second partition is fully consumed.
9 . The system of claim 8 , wherein the first claim is given over any available partition of the data stream group.
10 . The system of claim 8 , wherein the first period and the second period are each for a predefined length of time.
11 . The system of claim 8 , wherein the first period and the second period are each are defined by an expiration time.
12 . The system of claim 8 , wherein consuming the data in the first partition includes reading the data and sending the data to a database in parallel with the reading.
13 . The system of claim 12 , wherein the consumer waits until all the read data has been sent to the database prior to making a request for a claim over another available partition of the data stream group to continue.
14 . A non-transitory machine-readable storage medium that provides instructions that, if executed by a processor of a server executing a consumer cause operations to performed comprising:
transmitting a first partition claim request that asks for a first available partition of a plurality of partitions that belong to a same data stream group to consume, wherein each of the plurality of partitions stores a different portion of data that belongs to the same data stream group, and wherein the first partition claim request indicates an identifier of the data stream group;
receiving a first partition claim response that indicates a first grant of a first claim over a first partition of the plurality of partitions, wherein the first claim is valid for a first period, and wherein the first claim allows the consumer to consume data of the first partition during the first period, wherein the consumer is one of a plurality of consumers;
consuming, during the first period, the data in the first partition until the first claim expires or the data in the first partition is fully consumed;
responsive to fully consuming the data of the first partition prior to the first claim expiring, transmitting a second partition claim request that asks for a second available partition of the plurality of partitions, wherein after the first claim expires the first partition is available to be claimed by any of the plurality of consumers;
receiving a second partition claim response that indicates a second grant of a second claim over a second partition of the plurality of partitions, wherein the second claim is valid for a second period, and wherein the second claim allows this consumer to consume data of the second partition during the second period; and
consuming, during the second period, data in the second partition until the second claim expires or the data in the second partition is fully consumed.
15 . The non-transitory machine-readable storage medium of claim 14 , wherein the first claim is given over any available partition of the data stream group.
16 . The non-transitory machine-readable storage medium of claim 14 , wherein the first period and the second period are each for a predefined length of time.
17 . The non-transitory machine-readable storage medium of claim 14 , wherein the first period and the second period are each are defined by an expiration time.
18 . The non-transitory machine-readable storage medium of claim 14 , wherein consuming the data in the first partition includes reading the data and sending the data to a database in parallel with the reading.
19 . The non-transitory machine-readable storage medium of claim 18 , wherein the consumer waits until all the read data has been sent to the database prior to making a request for a claim over another available partition of the data stream group to continue.