Atomically committing related streaming data across multiple distributed resources
Implementations are provided herein for atomically committing related stream data across multiple, distributed resources. Transactions can be established that are distributed across multiple hosts, and their data can be made to appear atomic to an observing process. A master status for the transaction can be used to flag to other processes that the transaction is being committed. A stream to which the transaction is being appended to can be locked until the transaction data is committed in full. It can be appreciated that one an event to a transaction is acknowledged, and the transaction is merged, its commitment is guaranteed.
1. A method comprising:
generating a transaction associated with an active stream, wherein the transaction is appended to the active stream to enable data to be written to the active stream;
establishing a set of stream segments associated with the transaction, wherein event data associated with the set of stream segments is written and routed to the set of stream segments via the transaction;
dynamically receiving a set of events targeted to the transaction;
dynamically publishing events in the set of events into the set of stream segments, wherein the published events are only visible to a writer and become visible to one or more readers in response to a commit of the transaction; and
in response to receiving a request to merge the transaction:
sealing the transaction;
locking the active stream from at least one of being appended to or deletion;
committing the set of stream segments into the active stream, wherein committing the set of stream segments includes reattempting the committing the stream segment in response to committing a stream segment in the set of stream segments into the active stream fails; and
unlocking the active stream.
2. The method of claim 1 , wherein the set of stream segments is based on a set of active stream segments associated with the active stream.
3. The method of claim 2 , further comprising:
receiving a request to seal an active stream segment among the set of active stream segments; and
at least one of aborting the transaction or aborting the request to seal the active stream.
4. The method of claim 1 , further comprising:
in response to dynamically publishing events in the set of events, acknowledging the events to a client application.
5. The method of claim 1 , wherein committing the set of stream segment into the active stream includes setting a metadata bit indicating that the transaction is committing.
6. A system comprising at least one storage device and at least one hardware processor configured to:
generate a transaction associated with an active stream, wherein the transaction is appended to the active stream to enable data to be written to the active stream;
establish a set of stream segments associated with the transaction, wherein event data associated with the set of stream segments is written and routed to the set of stream segments via the transaction;
dynamically receive a set of events targeted to the transaction;
dynamically publish events in the set of events into the set of stream segments, wherein the published events are only visible to a writer and become visible to one or more readers in response to a commit of the transaction; and
in response to receiving a request to merge the transaction:
seal the transaction;
lock the active stream from at least one of being appended to or deletion;
commit the set of stream segments into the active stream, wherein committing the set of stream segments includes reattempting the committing the stream segment in response to committing a stream segment in the set of stream segments into the active stream fails; and
unlock the active stream.
7. The system of claim 6 , wherein the set of stream segments is based on a set of active stream segments associated with the active stream.
8. The system of claim 7 , further configured to:
receive a request to seal an active stream segment among the set of active stream segments; and
at least one of abort the transaction or abort the request to seal the active stream.
9. The system of claim 6 , further configured to:
in response to dynamically publishing events in the set of events, acknowledge the events to a client application.
10. The system of claim 6 , wherein committing the set of stream segment into the active stream includes setting a metadata bit indicating that the transaction is committing.
11. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
generating a transaction associated with an active stream, wherein the transaction is appended to the active stream to enable data to be written to the active stream;
establishing a set of stream segments associated with the transaction, wherein event data associated with the set of stream segments is written and routed to the set of stream segments via the transaction;
dynamically receiving a set of events targeted to the transaction;
dynamically publishing events in the set of events into the set of stream segments, wherein the published events are only visible to a writer and become visible to one or more readers in response to a commit of the transaction; and
in response to receiving a request to merge the transaction:
sealing the transaction;
locking the active stream from at least one of being appended to or deletion;
committing the set of stream segments into the active stream, wherein committing the set of stream segments includes reattempting the committing the stream segment in response to committing a stream segment in the set of stream segments into the active stream fails; and
unlocking the active stream.
12. The non-transitory computer readable medium of claim 11 , wherein the set of stream segments is based on a set of active stream segments associated with the active stream.
13. The non-transitory computer readable medium of claim 12 , with program instructions stored thereon to further perform the following acts:
receiving a request to seal an active stream segment among the set of active stream segments; and
at least one of aborting the transaction or aborting the request to seal the active stream.
14. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
in response to dynamically publishing events in the set of events, acknowledging the events to a client application.
15. The non-transitory computer readable medium of claim 11 , wherein committing the set of stream segment into the active stream includes setting a metadata bit indicating that the transaction is committing.