Executing streaming data writes without duplication or loss
Implementations are provided herein for executive streaming data writes without duplication or loss. A client application and a pravega node can work to track where write data is, how much data has been written, and what specific data has been acknowledged by the pravega node as successfully written. In the event of an error or connection disruption, the client application can reconnect and determine how much data has been written and resend what data still needs to be written. The data can be written exactly once, and once written and acknowledged, will no longer be subject to data loss.
1. A method comprising: packaging a set of raw stream data as an event by a client application, wherein the event is associated with an event time and a routing key, wherein the set of raw stream data includes unbounded data; causing each event to have serialized and deserialized access, enabling an application to use file-based storage read and write techniques on each event; using a hashing algorithm to assign a stream segment to the event based on the routing key; sending the event to a node; receiving the event by the node; storing the event in tier 1 storage based on the stream segment, the event time and the routing key; in response to successfully storing the event in tier 1 storage, associating the event in storage with the stream segment, the event time and the routing key; and sending an acknowledgement to the client application that the event was stored successfully.
2. The method of claim 1 , further comprising: in response to unsuccessfully storing the event in tier 1 storage, sending a notification to the client application that the event was not stored successfully.
3. The method of claim 2 , further comprising: resending the event to the node.
4. The method of claim 1 , wherein the routing key is based on at least one of the set of raw stream data, a unique user identifier, a sensor identifier, and a timestamp.
5. The method of claim 1 , wherein the hashing algorithm is a consistent hash.
6. A system comprising at least one storage device and at least one hardware processor configured to: package a set of raw stream data as an event by a client application, wherein the event is associated with an event time and a routing key, and wherein the set of raw stream data includes unbounded data; cause each event to have serialized and deserialized access, enabling an application to use file-based storage read and write techniques on each event, use a hashing algorithm to assign a stream segment to the event based on the routing key; send the event to a node; receive the event by the node; store the event in tier 1 storage based on the stream segment, the event time and the routing key; in response to successfully storing the event in tier 1 storage, associate the event in storage with the stream segment, the event time and the routing key; and send an acknowledgement to the client application that the event was stored successfully.
7. The system of claim 6 , further configured to: in response to unsuccessfully storing the event in tier 1 storage, send a notification to the client application that the event was not stored successfully.
8. The system of claim 7 , further configured to: resend the event to the node.
9. The system of claim 6 , wherein the routing key is based on at least one of the set of raw stream data, a unique user identifier, a sensor identifier, and a timestamp.
10. The system of claim 6 , wherein the hashing algorithm is a consistent hash.
11. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts: packaging a set of raw stream data as an event by a client application, wherein the event is associated with an event time and a routing key, and wherein the set of raw stream data includes unbounded data; causing each event to have serialized and deserialized access, enabling an application to use file-based storage read and write techniques on each event; using a hashing algorithm to assign a stream segment to the event based on the routing key; sending the event to a node; receiving the event by the node; storing the event in tier 1 storage based on the stream segment, the event time and the routing key; in response to successfully storing the event in tier 1 storage, associating the event in storage with the stream segment, the event time and the routing key; and sending an acknowledgement to the client application that the event was stored successfully.
12. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts: in response to unsuccessfully storing the event in tier 1 storage, sending a notification to the client application that the event was not stored successfully.
13. The non-transitory computer readable medium of claim 12 , with program instructions stored thereon to further perform the following acts: resending the event to the node.
14. The non-transitory computer readable medium of claim 11 , wherein the routing key is based on at least one of the set of raw stream data, a unique user identifier, a sensor identifier, and a timestamp.
15. The non-transitory computer readable medium of claim 11 , wherein the hashing algorithm is a consistent hash.