Stateless stream handling and resharding
Systems and methods are disclosed for stateless stream handling and resharding. In one implementation, a first shard including one or more messages is generated. The first shard is associated with a first shard version attribute. The first shard and the first shard version attribute are provided as a first atomic update within a data stream. The first shard is resharded into at least a second shard. The second shard is associated with a second shard version attribute. The second shard and the second shard version attribute are provided as a second atomic update within the data stream.
1. A system comprising:
a processing device; and
a memory coupled to the processing device and storing instructions that, when executed by the processing device, cause the system to perform operations comprising:
generating a first shard comprising one or more messages;
associating the first shard with a first shard version attribute;
providing the first shard and the first shard version attribute as a first atomic update within a data stream;
receiving a first state attribute that reflects a processing capacity of a streaming system;
based on the first state attribute that reflects the processing capacity of the streaming system, initiating a resharding of the first shard into at least a second shard that comprises at least one of the one or more messages from the first shard;
associating the second shard with a second shard version attribute; and
providing the second shard and the second shard version attribute as a second atomic update within the data stream.
2. The system of claim 1 , wherein resharding the first shard comprises generating the second shard based on the first shard.
3. The system of claim 1 , wherein the second shard comprises at least one of the one or more messages from the first shard.
4. The system of claim 1 , wherein the first state attribute reflects a processing capacity of a streaming system.
5. The system of claim 1 , wherein the first state attribute reflects a processing capacity of a consumer.
6. The system of claim 1 , wherein the memory further stores instructions to cause the system to perform operations comprising generating a third shard in accordance with the resharding.
7. A method comprising:
receiving, from a device, a first shard comprising one or more messages and a first shard version attribute;
requesting, from the device, a current shard version;
based on a determination that the current shard version is consistent with the first shard version attribute, performing an operation with respect to the first shard; and
based on a determination that the current shard version is consistent with the second shard version attribute, performing an operation with respect to the second shard in lieu of the operation with respect to the first shard.
8. The method of claim 7 , further comprising based on a determination that the current shard version is not consistent with the first shard version attribute, canceling performance of the operation with respect to the first shard.
9. The method of claim 7 , further comprising receiving, from the device, a second shard comprising a second shard version attribute.
10. The method of claim 7 , wherein performing an operation comprises providing the first shard to a consumer.
11. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:
receiving a first shard comprising one or more messages and a first shard version attribute;
requesting a current shard version;
based on a determination that the current shard version is consistent with the first shard version attribute, performing an operation with respect to the first shard;
based on a determination that the current shard version is consistent with the second shard version attribute, performing an operation with respect to the second shard in lieu of the operation with respect to the first shard.
12. The computer-readable medium of claim 11 , wherein the instructions further cause the processing device to perform operations comprising providing a resharding request.
13. The computer-readable medium of claim 12 , wherein the resharding request is provided based on a processing capacity of a device that received the first shard.
14. The computer-readable medium of claim 12 , wherein the resharding request is provided based on a processing capacity of another device.
15. The computer-readable medium of claim 11 , wherein the instructions further cause the processing device to perform operations comprising based on a determination that the current shard version is not consistent with the first shard version attribute, canceling performance of the operation with respect to the first shard.
16. The computer-readable medium of claim 11 , wherein the instructions further cause the processing device to perform operations comprising receiving a second shard comprising a second shard version attribute.
17. The computer-readable medium of claim 16 , wherein receiving a second shard comprises receiving the second shard in response to a resharding request.