Parallelism of a hyperscaler storage system
Write requests to store data at one or more solid-state storage devices are received by a storage controller from clients external to a storage system. A number of open segments in the storage system is identified. A determination as to whether to open one or more subsequent segments for the write requests is made based on the number of open segments and a target parallelism for the storage system. In response to determining that the one or more subsequent segments are to be opened based on the number of open segments and the target parallelism, the one or more subsequent segments are opened.
1 . A storage system comprising:
a plurality of non-volatile solid-state storage devices; and
a storage controller operatively coupled to the plurality of non-volatile solid-state storage devices, configured to:
receive, from a plurality of clients external to the storage system, a plurality of write requests to store data at one or more of the plurality of non-volatile solid-state storage devices;
identify a number of open segments within the storage system;
determine whether to open one or more subsequent segments for the plurality of write requests based on the number of open segments and a target parallelism for the storage system; and
in response to determining that the one or more subsequent segments are to be opened based on the number of open segments and the target parallelism, open the one or more subsequent segments.
2 . The storage system of claim 1 , wherein the storage controller is further configured to:
calculate a target number of open segments for processes of the storage system; and
determine whether a number of segments open for storage system processes is below the target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that the number of open segments for the storage system processes is below the target number of open segments.
3 . The storage system of claim 1 , wherein the storage controller is further configured to:
receive, from one or more of the plurality of clients, one or more subsequent write requests to store subsequent data at the one or more of the plurality of non-volatile solid-state storage devices;
determine whether to open or not open one or more new segments based on a subsequent number of open segments and the target parallelism for the storage system; and
in response to determining that the one or more new segments should not be opened based on the subsequent number of open segments and the target parallelism, bypass opening the one or more new segments.
4 . The storage system of claim 3 , wherein the storage controller is further configured to:
write the subsequent data from the one or more subsequent write requests to one or more segments that were previously opened.
5 . The storage system of claim 1 , wherein the plurality of non-volatile solid-state storage devices comprise erase blocks whose utilization is managed by the storage controller.
6 . The storage system of claim 1 , wherein the storage controller is further configured to:
calculate an allocation share for the plurality of write requests, wherein the calculation uses a ratio between a target ratio of open segments assigned to the one or more write requests, an aggregate of target ratios of open segments assigned to a plurality of storage system processes having open segments, and a target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that an open segment usage is under the allocation share.
7 . The storage system of claim 6 , wherein the allocation share for the one or more write requests is further calculated using a target ratio of open segments assigned to the one or more write requests and a target ratio of open segments assigned to other storage system processes having open segments.
8 . A method comprising:
receiving, by a storage controller from a plurality of clients external to a storage system, a plurality of write requests to store data at one or more of a plurality of non-volatile solid-state storage devices;
identifying a number of open segments within the storage system;
determining whether to open one or more subsequent segments for the plurality of write requests based on the number of open segments and a target parallelism for the storage system; and
in response to determining that the one or more subsequent segments are to be opened based on the number of open segments and the target parallelism, opening the one or more subsequent segments.
9 . The method of claim 8 , further comprising:
calculating a target number of open segments for processes of the storage system; and
determining whether a number of segments open for storage system processes is below the target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that the number of open segments for the storage system processes is below the target number of open segments.
10 . The method of claim 8 , further comprising:
receiving, from one or more of the plurality of clients, one or more subsequent write requests to store subsequent data at the one or more of the plurality of non-volatile solid-state storage devices;
determining whether to open or not open one or more new segments based on a subsequent number of open segments and the target parallelism for the storage system; and
in response to determining that the one or more new segments should not be opened based on the subsequent number of open segments and the target parallelism, bypassing opening the one or more new segments.
11 . The method of claim 10 , further comprising:
writing the subsequent data from the one or more subsequent write requests to one or more segments that were previously opened.
12 . The method of claim 8 , wherein the plurality of non-volatile solid-state storage devices comprise erase blocks whose utilization is managed by the storage controller.
13 . The method of claim 8 , further comprising:
calculating an allocation share for the plurality of write requests, wherein the calculation uses a ratio between a target ratio of open segments assigned to the one or more write requests, an aggregate of target ratios of open segments assigned to a plurality of storage system processes having open segments, and a target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that an open segment usage is under the allocation share.
14 . The method of claim 13 , wherein the allocation share for the one or more write requests is further calculated using a target ratio of open segments assigned to the one or more write requests and a target ratio of open segments assigned to other storage system processes having open segments.
15 . A non-transitory computer readable storage medium storing instructions which, when executed, cause a processing device of a storage controller to:
receive, from a plurality of clients external to a storage system, a plurality of write requests to store data at one or more of a plurality of non-volatile solid-state storage devices;
identify a number of open segments within the storage system;
determine whether to open one or more subsequent segments for the plurality of write requests based on the number of open segments and a target parallelism for the storage system; and
in response to determining that the one or more subsequent segments are to be opened based on the number of open segments and the target parallelism, open the one or more subsequent segments.
16 . The non-transitory computer readable storage medium of claim 15 , wherein the storage controller is further to:
calculate a target number of open segments for processes of the storage system; and
determine whether a number of segments open for storage system processes is below the target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that the number of open segments for the storage system processes is below the target number of open segments.
17 . The non-transitory computer readable storage medium of claim 15 , wherein the storage controller is further to:
receive, from one or more of the plurality of clients, one or more subsequent write requests to store subsequent data at the one or more of the plurality of non-volatile solid-state storage devices;
determine whether to open or not open one or more new segments based on a subsequent number of open segments and the target parallelism for the storage system; and
in response to determining that the one or more new segments should not be opened based on the subsequent number of open segments and the target parallelism, bypass opening the one or more new segments.
18 . The non-transitory computer readable storage medium of claim 17 , wherein the storage controller is further to:
write the subsequent data from the one or more subsequent write requests to one or more segments that were previously opened.
19 . The non-transitory computer readable storage medium of claim 15 , wherein the plurality of solid-state storage devices comprise erase blocks whose utilization is managed by the storage controller.
20 . The non-transitory computer readable storage medium of claim 15 , wherein the storage controller is further to:
calculate an allocation share for the plurality of write requests, wherein the calculation uses a ratio between a target ratio of open segments assigned to the one or more write requests, an aggregate of target ratios of open segments assigned to a plurality of storage system processes having open segments, and a target number of open segments, wherein the subsequent segment is opened for the one or more write requests in response to determining that an open segment usage is under the allocation share.