STATELESS STREAM HANDLING AND RESHARDING
Systems and methods are disclosed for stateless stream handling and resharding. In one implementation, a first shard comprising one or more messages is received. The first shard is associated with a first state attribute. The first shard and the first state attribute are provided as an update within a data stream. In another implementation, a first shard including one or more messages is received within a first stream. The first shard is associated with a first state attribute. The first shard and the first state attribute are provided as an update within a data stream.
1 . A system comprising:
a processing device; and
a memory coupled to the processing device and storing instructions that, when executed by the processing device, cause the system to perform operations comprising:
receiving a first shard comprising one or more messages;
associating the first shard with a first state attribute; and
providing the first shard and the first state attribute as an atomic update within a data stream.
2 . The system of claim 1 , wherein the memory further stores instructions to cause the system to perform operations comprising requesting the first state attribute from the first shard.
3 . The system of claim 2 , wherein the memory further stores instructions to cause the system to perform operations comprising:
receiving the first state attribute; and
providing a second shard within the data stream based on the received first state attribute.
4 . The system of claim 1 , wherein the first state attribute reflects an importance of one or more of the messages.
5 . The system of claim 1 , wherein the first state attribute reflects a location of one or more of the messages.
6 . The system of claim 1 , wherein the memory further stores instructions to cause the system to perform operations comprising:
receiving the first state attribute; and
initiating an adjustment of an operation of a message production source based on the received first state attribute.
7 . The system of claim 6 , wherein the first state attribute comprises a token assigned based on a processing capacity of a streaming system.
8 . The system of claim 1 , wherein an operation associated with the first shard is performed based on the first state attribute.
9 . The system of claim 1 , wherein the atomic update comprises a plurality of updates that are collectively performed or rejected.
10 . The system of claim 1 , wherein providing the first shard and the first state attribute comprises providing the first shard and the first state attribute as a conditional update within the data stream.
11 . The system of claim 1 , wherein the first state attribute comprises a first sequence identifier.
12 . A method comprising:
receiving a first shard comprising one or more messages;
associating the first shard with a first state attribute; and
providing the first shard and the first state attribute as an update within a data stream.
13 . The method of claim 12 , further comprising requesting the first state attribute from the first shard.
14 . The method of claim 13 , further comprising:
receiving the first state attribute; and
providing a second shard within the data stream based on the received first state attribute.
15 . The method of claim 12 , wherein the first state attribute reflects an importance of one or more of the messages.
16 . The method of claim 12 , wherein the first state attribute reflects a location of one or more of the messages.
17 . The method of claim 12 , wherein providing the first shard and the first state attribute comprises providing the first shard and the first state attribute as an atomic update within the data stream.
18 . The method of claim 12 , wherein providing the first shard and the first state attribute comprises providing the first shard and the first state attribute as a conditional update within the data stream.
19 . The method of claim 12 , wherein the first state attribute comprises a first sequence identifier.
20 . A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:
receiving a first shard comprising one or more messages;
associating the first shard with a first sequence identifier;
associating the first shard with a first state attribute that reflects an importance of one or more of the messages; and
providing the first shard and the first state attribute as an update within a data stream;
requesting the first state attribute from the first shard;
receiving the first state attribute; and
providing a second shard within the data stream based on the received first state attribute.