Efficiently executing concurrent operations that depend on each other in a streaming data environment
Implementations are provided herein for accepting operations asynchronously in a particular order and efficiently committing them into an append-only log while preserving relative order. Operations that are dependent on one or more operations prior to it in the log will be guaranteed to fail, and not accepted, if any of the prior operations failed. If an operation succeeds, it is guaranteed that all operations it depended on are also successful.
1. A method comprising:
dynamically receiving streaming data operations;
dynamically packaging streaming data operations into a set of sequential frames;
dynamically generating a metadata update transaction (“MUT”) associated with each frame in the sequential set of frames;
asynchronously processing a frame in the sequential set of frames by:
validating the streaming data operations of the frame against the MUT associated with the frame;
serializing operations of the frame;
sealing the frame;
sealing the MUT associated with the frame; and
sending the first frame to Tier 1 storage.
2. The method of claim 1 , wherein operations include at least one of create segment, delete segment, append to segment, seal segment, or merge segments.
3. The method of claim 1 , further comprising:
asynchronously receiving acknowledgement from Tier 1 storage that frames in the set of sequential frames have been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was stored successfully:
determining a sequential placement of the frame in the set of sequential frames; and
processing any unprocessed MUT's associated with the frame and frames in the set of sequential frames earlier than the sequential placement of the frame.
4. The method of claim 3 , wherein processing an MUT includes updating a global metadata store based on the MUT.
5. The method of claim 1 , further comprising:
asynchronously receiving notification from Tier 1 storage that frames in the set of sequential frames have not been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was not stored successfully:
determining a sequential placement of the frame in the set of sequential frames;
discarding any MUT's associated with the frame and frames in the set of sequential frames later than the sequential placement of the frame; and
notifying a caller of a failure associated with the frame.
6. A system comprising at least one storage device and at least one hardware processor configured to:
dynamically receive streaming data operations;
dynamically package streaming data operations into a set of sequential frames;
dynamically generate a metadata update transaction (“MUT”) associated with each frame in the sequential set of frames;
asynchronously process a frame in the sequential set of frames by:
validating the streaming data operations of the frame against the MUT associated with the frame;
serializing operations of the frame;
sealing the frame;
sealing the MUT associated with the frame; and
sending the first frame to Tier 1 storage.
7. The system of claim 6 , wherein operations include at least one of create segment, delete segment, append to segment, seal segment, or merge segments.
8. The system of claim 6 , further configured to:
asynchronously receive acknowledgement from Tier 1 storage that frames in the set of sequential frames have been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was stored successfully:
determine a sequential placement of the frame in the set of sequential frames; and
process any unprocessed MUT's associated with the frame and frames in the set of sequential frames earlier than the sequential placement of the frame.
9. The system of claim 8 , wherein processing an MUT includes updating a global metadata store based on the MUT.
10. The system of claim 6 , further configured to:
asynchronously receive notification from Tier 1 storage that frames in the set of sequential frames have not been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was not stored successfully:
determine a sequential placement of the frame in the set of sequential frames;
discard any MUT's associated with the frame and frames in the set of sequential frames later than the sequential placement of the frame; and
notify a caller of a failure associated with the frame.
11. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
dynamically receiving streaming data operations;
dynamically packaging streaming data operations into a set of sequential frames;
dynamically generating a metadata update transaction (“MUT”) associated with each frame in the sequential set of frames;
asynchronously processing a frame in the sequential set of frames by:
validating the streaming data operations of the frame against the MUT associated with the frame;
serializing operations of the frame;
sealing the frame;
sealing the MUT associated with the frame; and
sending the first frame to Tier 1 storage.
12. The non-transitory computer readable medium of claim 11 , wherein operations include at least one of create segment, delete segment, append to segment, seal segment, or merge segments.
13. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
asynchronously receiving acknowledgement from Tier 1 storage that frames in the set of sequential frames have been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was stored successfully:
determining a sequential placement of the frame in the set of sequential frames; and
processing any unprocessed MUT's associated with the frame and frames in the set of sequential frames earlier than the sequential placement of the frame.
14. The non-transitory computer readable medium of claim 13 , wherein processing an MUT includes updating a global metadata store based on the MUT.
15. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
asynchronously receiving notification from Tier 1 storage that frames in the set of sequential frames have not been stored successfully; and
in response to receive acknowledgement that a frame in the set of sequential frames was not stored successfully:
determining a sequential placement of the frame in the set of sequential frames;
discarding any MUT's associated with the frame and frames in the set of sequential frames later than the sequential placement of the frame; and
notifying a caller of a failure associated with the frame.