Data storage management and resource scaling
Aspects of the present disclosure relate to managing data storage resources. In embodiments, one or more data streams are received. Each data stream can include one or more data portions. Further, one or more storage parameters are monitored. Each storage parameter can include data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices. Data storage resources are elastically scales based on any changes to the at least one monitored storage parameter.
1. An apparatus configured to a memory and at least one processor configured to:
receive one or more data streams, wherein each data stream includes one or more data portions;
monitor one or more storage parameters, wherein each storage parameter corresponds to one or more of: data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices; and
elastically scale data storage resources based on any changes to the at least one monitored storage parameter, wherein elastically scaling the storage resources includes:
monitoring at least one stream's data input rate;
determining the at least one stream's service level objective (SLO); and
adjusting a number of readers in a readers in a reader group configred to process data from the stream based on the data input rate and SLO.
2. The apparatus of claim 1 further configured to shard each data portion into one or more stream segments based on at least one of the monitored storage parameters.
3. The apparatus of claim 2 , wherein each sharded stream segment corresponds to a container configured to buffer one or more sets of data portions and each sharded stream segment corresponds to one of the data storage resources.
4. The apparatus of claim 2 further configured to write each data portion to at least one of the sharded stream segments based on a routing key corresponding to each data portion.
5. The apparatus of claim 2 , wherein each data portion corresponds to an event and each event represents a data record.
6. The apparatus of claim 5 further configured to group one or more readers into one or more sets of reader groups based on one or more of: a number of sharded stream segments and the at least one monitored storage parameter, wherein each reader and each reader group corresponds to two of the data storage resources.
7. The apparatus of claim 6 further configured to assign each reader from each reader group to zero or more of the sharded stream segments.
8. The apparatus of claim 7 further configured to elastically scale an amount of the sharded stream segments based on the at least one monitored storage parameter.
9. The apparatus of claim 8 further configured to adjust elastically scaling an amount of one or more of the readers in each reader group, an amount of reader groups, and an amount of readers assigned to each of the scaled amount of sharded stream segments.
10. The apparatus of claim 1 , wherein elastically scaling the data storage resources includes one or more of growing or shrinking an amount of the data storage resources.
11. A method comprising:
receiving one or more data streams, wherein each data stream includes one or more data portions;
monitoring one or more storage parameters, wherein each storage parameter corresponds to one or more of: data input load of each data stream, data write rate of each stream, number of events in each stream, and data ingestion rates of one or more storage devices; and
elastically scaling data storage resources based on any changes to the at least one monitored storage parameter, wherein elastically scaling the storage resources includes:
monitoring at least one stream's data input rate;
determining at least one stream's service level objective (SLO); and
adjusting a number of readers in a readers in a reader group configred to process data from the stream based on the data input rate and SLO.
12. The method of claim 11 further comprising sharding each data portion into one or more stream segments based on at least one of the monitored storage parameters.
13. The method of claim 12 , wherein each sharded stream segment corresponds to a container configured to buffer one or more sets of data portions and each sharded stream segment corresponds to one of the data storage resources.
14. The method of claim 12 further comprising writing each data portion to at least one of the sharded stream segments based on a routing key corresponding to each data portion.
15. The method of claim 12 , wherein each data portion corresponds to an event and each event represents a data record.
16. The method of claim 15 further comprising grouping one or more readers into one or more sets of reader groups based on one or more of: a number of sharded stream segments and the at least one monitored storage parameter, wherein each reader and each reader group corresponds to two of the data storage resources.
17. The method of claim 16 further comprising assigning each reader from each reader group to zero or more of the sharded stream segments.
18. The method of claim 17 , wherein elastically scaling the data storage resources includes elastically scaling an amount of the sharded stream segments based on the at least one monitored storage parameter.
19. The method of claim 18 further comprising adjusting elastically scaling an amount of one or more of the readers in each reader group, an amount of reader groups, and an amount of readers assigned to each of the scaled amount of sharded stream segments.
20. The method of claim 11 , wherein elastically scaling the data storage resources includes one or more of growing or shrinking an amount of the data storage resources.