Bypassing of tier-1 storage in a streaming data storage system
The described technology is generally directed towards a streaming data storage system that can switch between a tiered mode of operation in which events are written to Tier-1 storage and later migrated to Tier-2 storage, and a direct mode of operation in which events are written to Tier-2 storage, bypassing the tiered mode. The switching from tiered mode to direct mode, and from direct mode to tiered mode, can be automatic and based on user configuration information. For example, an event size metric (e.g., average event size) can be evaluated against user defined thresholds to determine which mode to use. If the average event size goes below a low threshold value, the tiered mode is switched to and used for appending events to a segment of a data stream. If the average event size goes above a high threshold value, the direct mode is switched to and used.
1. A system, comprising:
a processor; and
a memory that stores executable instructions which, in response to being executed by the processor of the system, facilitate performance of operations, the operations comprising:
determining whether to switch a current storage mode for a segment of an event stream between a tiered mode and a direct mode, wherein the tiered mode comprises storing received events of the segment in a first tier data storage, concatenating the received events into concatenated data based on a sequential order in which the events were received, and moving the concatenated data to a second tier data storage, wherein the direct mode comprises storing the received events of the segment directly in the second tier data storage based on a sequential order in which the events were received, wherein the first tier data storage comprises a first storage device selected from a first group of storage devices comprising a solid-state drive device and a non-volatile memory express storage device, wherein the second tier data storage comprises a second storage device selected from a second group of storage devices comprising a hard drive device, and wherein the determining comprises:
in response to the current storage mode being the tiered mode:
determining whether a first average event size of respective sizes of a defined quantity of most recent events of the segment stored in the first tier data storage exceeds a first threshold size, and
in response to the first average event size exceeding the first threshold size changing the current storage mode to the direct mode for subsequent events of the segment; and
in response to the current storage mode being the direct mode:
determining whether a second average event size of respective sizes of a defined quantity of most recent events of the segment stored in the second tier data storage is less than a second threshold size, wherein the second threshold size is less than the first threshold size, and
in response to the second average event size is less than the second threshold size changing the current storage mode to the tiered mode for subsequent events of the segment.
2. The system of claim 1 , wherein the changing the current storage mode to the direct mode comprises delaying storage of the subsequent events of the segment to the second tier data storage until after all events of the segment in the first tier data storage have been moved to the second tier data storage.
3. The system of claim 1 , wherein the first threshold size and the second threshold size are configurable based on user input.
4. The system of claim 1 , wherein the first threshold size and the second threshold size are determined based on a threshold value and a sensitivity value.
5. The system of claim 1 , wherein the defined quantity is based on a percentage of events.
6. The system of claim 1 , wherein the defined quantity is based on a defined time period.
7. The system of claim 1 , wherein the changing the current storage mode to the tiered mode comprises delaying movement of concatenated data of the subsequent events from the first tier data storage to the second tier data storage until after all delayed events of the segment that were being directly being stored to the second tier data storage have been completed.
8. The system of claim 7 , wherein the first threshold size and the second threshold size selected based on a performance criterion with respect to storage of the segment of the event stream.
9. The system of claim 1 , wherein the concatenated data moved from the first tier data storage is stored in different chunks from events of the segment directly stored in the second tier data storage.
10. The system of claim 9 , wherein the chunks containing the events of the segment comprise respective metadata that maintain a sequential order in which the events of the segment were received.
11. A method, comprising:
determining, by a system comprising a processor, whether to switch a current storage mode for a segment of an event stream between a tiered mode and a direct mode, wherein the tiered mode comprises storing received events of the segment in a first tier data storage, concatenating the received events into concatenated data based on a sequential order in which the events were received, and moving the concatenated data to a second tier data storage, wherein the direct mode comprises storing the received events of the segment directly in the second tier data storage based on a sequential order in which the events were received, wherein the second tier data storage comprises a hard drive device, wherein the first tier data storage comprises a solid-state drive device or a non-volatile memory express storage device, and wherein the determining comprises:
in response to the current storage mode being the tiered mode:
determining, by the system, whether a first average event size of respective sizes of a defined quantity of most recent events of the segment stored in the first tier data storage exceeds a first threshold size, and
in response to the first average event size exceeding the first threshold size changing, by the system, the current storage mode to the direct mode for subsequent events of the segment; and
in response to the current storage mode being the direct mode:
determining, by the system, whether a second average event size of respective sizes of a defined quantity of most recent events of the segment stored in the second tier data storage is less than a second threshold size, wherein the second threshold size is less than the first threshold size, and
in response to the second average event size is less than the second threshold size changing, by the system, the current storage mode to the tiered mode for subsequent events of the segment.
12. The method of claim 11 , wherein the changing the current storage mode to the direct mode comprises delaying storage of the subsequent events of the segment to the second tier data storage until after all events of the segment in the first tier data storage have been moved to the second tier data storage.
13. The method of claim 11 , wherein the changing the current storage mode to the tiered mode comprises delaying movement of concatenated data of the subsequent events from the first tier data storage to the second tier data storage until after all delayed events of the segment that were being directly being stored to the second tier data storage have been completed.
14. The method of claim 11 , wherein the defined quantity is based on a percentage of events.
15. The method of claim 11 , wherein the defined quantity is based on a defined time period.
16. A non-transitory machine-readable medium, comprising executable instructions that, in response to being executed by a processor of a streaming data storage system, facilitate performance of operations, the operations comprising:
determining whether to switch a current storage mode for a segment of an event stream between a tiered mode and a direct mode, wherein the tiered mode comprises storing received events of the segment in a first tier data storage, concatenating the received events into concatenated data based on a sequential order in which the events were received, and moving the concatenated data to a second tier data storage, and wherein the direct mode comprises storing the received events of the segment directly in the second tier data storage based on a sequential order in which the events were received, wherein the first tier data storage comprises a first storage device selected from a first group of storage devices comprising a solid-state drive device or a non-volatile memory express storage device, and wherein the first tier data storage comprises a second storage device selected from a second group of storage devices comprising a hard drive storage device, and wherein the determining comprises:
in response to the current storage mode being the tiered mode:
determining whether a first average event size of respective sizes of a defined quantity of most recent events of the segment stored in the first tier data storage exceeds a first threshold size, and
in response to the first average event size exceeding the first threshold size changing the current storage mode to the direct mode for subsequent events of the segment; and
in response to the current storage mode being the direct mode:
determining whether a second average event size of respective sizes of a defined quantity of most recent events of the segment stored in the second tier data storage is less than a second threshold size, wherein the second threshold size is less than the first threshold size, and
in response to the second average event size is less than the second threshold size changing the current storage mode to the tiered mode for subsequent events of the segment.
17. The non-transitory machine-readable medium of claim 16 , wherein the changing the current storage mode to the direct mode comprises delaying storage of the subsequent events of the segment to the second tier data storage until after all events of the segment in the first tier data storage have been moved to the second tier data storage.
18. The non-transitory machine-readable medium of claim 16 , wherein the changing the current storage mode to the tiered mode comprises delaying movement of concatenated data of the subsequent events from the first tier data storage to the second tier data storage until after all delayed events of the segment that were being directly being stored to the second tier data storage have been completed.
19. The non-transitory machine-readable medium of claim 16 , wherein the concatenated data moved from the first tier data storage is stored in different chunks from events of the segment directly stored in the second tier data storage.
20. The non-transitory machine-readable medium of claim 19 , wherein the chunks containing the events of the segment comprise respective metadata that maintain a sequential order in which the events of the segment were received.