Live resegmenting of partitions in distributed stream-processing platforms

1. A non-transitory computer readable medium comprising instructions which, when executed by one or more hardware processors, causes performance of operations comprising:

upon receiving a trigger to move a partition of a distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources, allocating the partition on the second broker;

configuring the first broker to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker; and

updating metadata for processing requests for the partition to include the second broker.

2. The medium of claim 1 , wherein the operations further comprise:

merging idempotent produce metadata for the partition from the first broker into the second broker after the first broker is configured to redirect the requests for the new messages after the last offset in the partition to the second broker.

3. The medium of claim 2 , wherein the operations further comprise:

using the idempotent produce metadata to validate, at the first broker, write requests for the partition prior to merging the idempotent produce data from the first broker into the second broker.

4. The medium of claim 2 , wherein the idempotent produce metadata comprises:

a producer identifier for a producer; and

a latest sequence number for the producer.

5. The medium of claim 1 , wherein the operations further comprise:

configuring the first broker to process read requests for old messages before the last offset in the partition during a retention period for the old messages.

6. The medium of claim 1 , wherein the operations further comprise:

using the metadata to direct read and write requests for the partition to the first and second brokers.

7. The medium of claim 1 , wherein configuring the first broker to redirect the requests for the new messages after the last offset in the partition to the second broker comprises:

updating the first broker with a redirect state and a redirect destination representing the second broker.

8. The medium of claim 1 , wherein redirecting the requests for the new messages after the last offset in the partition to the second broker comprises:

transmitting, with a request redirected to the second broker, the last offset in the partition for use in setting a base offset for the partition in the second broker.

9. The medium of claim 1 , wherein updating the metadata for processing requests for the partition to include the second broker comprises:

using a stream in the distributed stream-processing platform to propagate the metadata to a set of interface nodes in the distributed stream-processing platform.

10. The medium of claim 1 , wherein the trigger is received in response to a change in load on the first broker.

11. The medium of claim 1 , wherein the metadata further comprises a cluster, a topic, and the last offset.

12. A method, comprising:

upon receiving a trigger to move a partition of a distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources, allocating the partition on the second broker;

configuring, by a computer system, the first broker to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker; and

updating metadata for processing requests for the partition to include the second broker.

13. The method of claim 12 , further comprising:

merging idempotent produce metadata for the partition from the first broker into the second broker after the first broker is configured to redirect the requests for the new messages after the last offset in the partition to the second broker; and

using the idempotent produce metadata to validate, at the first broker, write requests for the partition prior to merging the idempotent produce data from the first broker into the second broker.

14. The method of claim 13 , wherein the idempotent produce metadata comprises:

a producer identifier for a producer; and

a latest sequence number for the producer.

15. The method of claim 12 , further comprising:

configuring the first broker to process read requests for old messages before the last offset in the partition during a retention period for the old messages.

16. The method of claim 12 , further comprising:

using the metadata to direct read and write requests for the partition to the first and second brokers.

17. The method of claim 12 , wherein configuring the first broker to redirect the requests for the new messages after the last offset in the partition to the second broker comprises:

updating the first broker with a redirect state and a redirect destination representing the second broker.

18. The method of claim 12 , wherein redirecting the requests for the new messages after the last offset in the partition to the second broker comprises:

transmitting, with a request redirected to the second broker, the last offset in the partition for use in setting a base offset for the partition in the second broker.

19. An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to:

upon receiving a trigger to move a partition of a distributed stream-processing platform from a first broker on a first set of physical resources to a second broker on a second a set of physical resources, allocate the partition on the second broker;

configure the first broker to redirect, to the second broker, requests for new messages after a last offset in the partition without replicating older messages before the last offset to the second broker; and

update metadata for processing requests for the partition to include the second broker.

20. The apparatus of claim 19 , wherein the memory further stores instructions that, when executed by the one or more processors, cause the apparatus to:

merge idempotent produce metadata for the partition from the first broker into the second broker after the first broker is configured to redirect the requests for the new messages after the last offset in the partition to the second broker.