Ordered event stream merging
Merging of portions of ordered event streams is disclosed. The disclosed merging of events can limit loss of order of events from streams in exchange for reduced computational load by grouping events according to a pseudo-epoch, wherein events are stored according to a scheme, even though the grouping can result reading events in a different order that that in which the events were written. However, by grouping the events, there can be fewer transitions between storage schemes when reading events than if they were read in the same order in which they were written, thereby reducing computational load. Moreover, restraints on the loss of order can be imposed by selecting a maximum progress window and generally restricting groups from comprising events two different storage schemes. Where events can be moved to archival storage, reducing storage scheme transitions can be of further benefit and speed access times of archived events.
1. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
receiving an indication to merge a first portion of a first ordered event stream and a second portion of a second ordered event stream, wherein the first ordered event stream and the second ordered event stream are stored via an ordered event stream storage system;
determining a read order of events of the first portion of the first ordered event stream and the second portion of the second ordered event stream, wherein the read order supports out of order reading of an event, and wherein the out of order reading is a different order than a first writing order of events of the first event stream and a second writing order of events of the second event stream; and
enabling reading of the events of the first portion of the first ordered event stream and the second portion of the second ordered event stream according to the read order.
2. The system of claim 1 , wherein the read order is preserved to facilitate future reads of the first portion of the first ordered event stream and the second portion of the second ordered event stream according to the read order.
3. The system of claim 1 , wherein the events of the first portion of the first ordered event stream and the second portion of the second ordered event stream are rewritten to reduce seek times corresponding to reading the events according to the read order.
4. The system of claim 1 , wherein the first ordered event stream and the second ordered event stream are a same ordered event stream.
5. The system of claim 1 , wherein the first ordered event stream and the second ordered event stream are different ordered event streams.
6. The system of claim 5 , wherein the first portion of the first ordered event stream corresponds to a first event storage scheme, wherein the second portion of the second ordered event stream corresponds to a second event storage scheme, and wherein the first event storage scheme and the second event storage scheme are a same event storage scheme.
7. The system of claim 5 , wherein the first portion of the first ordered event stream corresponds to a first event storage scheme, wherein the second portion of the second ordered event stream corresponds to a second event storage scheme, and wherein the first event storage scheme and the second event storage scheme are different event storage schemes.
8. The system of claim 7 , wherein the operations further comprise:
determining a first pseudo-epoch comprising first events of the first portion of the first ordered event stream;
determining a second pseudo-epoch comprising second events of the second portion of the second ordered event stream; and
wherein the determining a read order of events of the first portion of the first ordered event stream and the second portion of the second ordered event stream comprises determining a progress step read order of the first events and the second events according to the first pseudo-epoch and the second pseudo-epoch.
9. The system of claim 8 , wherein the first pseudo-epoch does not comprise events from more than one epoch of the first portion of the first ordered event stream, and wherein the second pseudo-epoch does not comprise events from more than one epoch of the second portion of the second ordered event stream.
10. The system of claim 8 , wherein the first pseudo-epoch does not exceed a first maximum progress window size, and wherein the second pseudo-epoch does not exceed a second maximum progress window size.
11. The system of claim 8 , wherein the determining the progress step read order of the first events and the second events according to the first pseudo-epoch and the second pseudo-epoch results in reading the events of the first portion of the first ordered event stream and the second portion of the second ordered event stream in an order that is a different order than the events were written to the first portion of the first ordered event stream and the second portion of the second ordered event stream.
12. The system of claim 8 , wherein the determining the progress step read order of the first events and the second events according to the first pseudo-epoch and the second pseudo-epoch results in reading the events of the first portion of the first ordered event stream and the second portion of the second ordered event stream with less vacillation between the first event storage scheme and the second event storage scheme than would be experienced where the events of the first portion of the first ordered event stream and the second portion of the second ordered event stream are read in the order than the events were written to the first portion of the first ordered event stream and the second portion of the second ordered event stream.
13. The system of claim 1 , wherein the first portion of the first ordered event stream comprises first events stored in a first type of data storage and comprises second events stored in a second type of data storage.
14. The system of claim 13 , wherein the first type of data storage enables faster access to data than the second type of data storage.
15. A method, comprising:
determining, by a system comprising a processor, a read order of merged events of a first portion of a first ordered event stream and a second portion of a second ordered event stream, wherein the first ordered event stream and the second ordered event stream are stored via an ordered event stream storage system, wherein the read order of the merged events comprises an out of order event read in comparison to a writing order of events into the first event stream and the second event stream, wherein the first ordered event stream and the second ordered event stream are different ordered event streams, wherein the first ordered event stream employs a first event storage scheme corresponding to a first epoch, wherein the second ordered event stream employs a second event storage scheme corresponding to a second epoch, and wherein the read order is based on a progress step associated with a first pseudo-epoch corresponding to first events of the first portion of the first epoch and a second pseudo-epoch corresponding to second events of the second portion of the second epoch; and
reading, by the system, the first events of the first portion of the first epoch and the second events of the second portion of the second epoch according to the read order of the merged events.
16. The method of claim 15 , further comprising performing, by the system, an operation selected from a group of operations comprising a storing operation that stores the read order of merged events to facilitate future reads of the merged events, and a relocation operation that relocates the first events and the second events in the ordered event stream storage system to reduce seek times associated with reading the merged events.
17. The method of claim 15 , wherein the determining the read order of the merged events comprises determining the first pseudo-epoch and the second pseudo-epoch, wherein the first and second pseudo-epoch are each smaller than a selectable maximum progress size, wherein the first and second pseudo-epoch each do not transition an epoch boundary, and wherein the first and second pseudo-epoch each comprise a same progress point.
18. A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising:
in response to receiving a merge events instruction, determining a read order of merged events of a first portion of a first ordered event stream and a second portion of a second ordered event stream, wherein:
the read order of the merged events corresponds to reading events in a different order than another order in which those events were written into the first event stream and the second event stream,
the first ordered event stream and the second ordered event stream are stored via an ordered event stream storage system,
the first ordered event stream and the second ordered event stream are different ordered event streams,
the first ordered event stream employs a first event storage scheme corresponding to a first epoch,
the second ordered event stream employs a second event storage scheme corresponding to a second epoch, and
the read order is based on a progress step associated with a first pseudo-epoch corresponding to first events of the first portion of the first epoch and a second pseudo-epoch corresponding to second events of the second portion of the second epoch; and
enabling reading of the first events of the first portion of the first epoch and the second events of the second portion of the second epoch according to the read order of the merged events, wherein the read order of the merged events is different than the write order of events comprising the merged events, and wherein the read order of the merged events reduces transitions between the first event storage scheme and the second event storage scheme in comparison to reading the merged events in the order they were written to the ordered event stream storage system.
19. The non-transitory machine-readable medium of claim 18 , wherein the first and second pseudo-epochs are smaller than a maximum progress window size, and wherein the first and second pseudo-epoch do not transition an epoch boundary.
20. The non-transitory machine-readable medium of claim 18 , wherein the first portion of the first ordered event stream comprises first events stored in a first type of data storage and comprises second events stored in a second type of data storage that is a different type of data storage than the first type of data storage.