FLEXIBLE RAID DRIVE GROUPING BASED ON PERFORMANCE
Systems and methods for RAID data storage in which data is written across a subset of the RAID drives, where the subset is selected based on drive performance. For instance, if a write will use N−2 of a total of N drives, the system may be configured to determine the two most heavily loaded drives (e.g., based on the respective weighted queue depths of the drives), and may exclude these drives from the write. The data may then be written to the remaining N−2 drives. The system may be configured to determine the RAID encoding for each write request independently of other writes, so the number of drives which are excluded may vary between write requests.
1 . A system for RAID storage of data, the system comprising:
a plurality of storage drives; and
a storage engine coupled to the storage drives;
wherein the storage engine is configured to receive from a user a plurality of write requests to write data on one or more of the plurality of storage drives;
wherein for each of one or more of the plurality of write requests, the storage engine is configured to
determine a corresponding RAID encoding;
determine a number, M, of the plurality of storage drives required for the corresponding RAID encoding, wherein the number is less than a total number, N, of the plurality of storage drives;
determine expected performance for one or more of the plurality of storage drives;
select a set of storage drives including M of the plurality of storage drives based at least in part on the expected performance for the plurality of storage drives
write corresponding data to the selected set of M storage drives using the corresponding RAID encoding.
2 . The system of claim 1 , wherein the storage engine is configured to:
determine the expected performance by determining a queue depth for each of the N storage drives; and
select the set of M storage drives by excluding one or more of the storage drives which have a lowest expected performance of the N storage drives and selecting the set of M storage drives from a remainder of the storage drives.
3 . The system of claim 2 , wherein the queue depth for each of the plurality of storage drives comprises a sum of a number of pending reads and a number of pending writes.
4 . The system of claim 2 , wherein the queue depth for each of the plurality of storage drives comprises a sum of a weighted number of pending reads and a weighted number of pending writes.
5 . The system of claim 2 , wherein a first one of the one or more of the plurality of write requests is written to a first set of M storage drives and a second one of the one or more of the plurality of write requests is written to a second set of M storage drives which is different from the first set of M storage drives.
6 . The system of claim 1 , wherein the storage engine is further configured to determine the RAID encoding corresponding to each write request based at least in part on a service level indicated by the user, wherein the service level includes a redundancy level.
7 . The system of claim 1 , wherein the storage engine is further configured to determine the RAID encoding corresponding to each write request based at least in part on a service level indicated by the user, wherein the service level includes a data access speed.
8 . The system of claim 1 , wherein the storage engine is configured to maintain a metadata tree, wherein for each write request, the metadata tree includes a corresponding entry wherein a key of the entry comprises the user address and a value of the entry comprises the one or more corresponding physical addresses.
9 . The system of claim 8 , wherein for each write request, the key of the corresponding entry further comprises a data length, and the value of the corresponding entry further comprises the RAID encoding.
10 . A system for RAID storage of data, the system comprising:
a plurality of storage drives; and
a storage engine coupled to the storage drives;
wherein the storage engine is configured to receive from a user a plurality of write requests to write data on one or more of the plurality of storage drives;
wherein for each of one or more of the plurality of write requests, the storage engine is configured to
determine an expected performance for each of the plurality of storage drives;
determine a number, M, of the plurality of storage drives that have expected performance which have an acceptable level of expected performance, wherein M is less than a total number, N, of the storage drives;
determine, based at least in part on the number, M, of the plurality of storage drives, a corresponding RAID encoding;
select a set of storage drives including M of the plurality of storage drives having the acceptable level of expected performance based at least in part on the corresponding RAID encoding
write corresponding data to the one or more selected storage drives using the corresponding RAID encoding.
11 . The system of claim 11 , wherein the storage engine is configured to determine the expected performance by:
determining a queue depth for each of the N storage drives, the queue depth comprising a sum of a weighted number of pending reads and a weighted number of pending writes; and
select the set of M storage drives by excluding one or more of the storage drives which have a lowest expected performance of the N storage drives and selecting the set of M storage drives from a remainder of the storage drives.
12 . The system of claim 10 , wherein the storage engine is further configured to determine the RAID encoding corresponding to each write request based at least in part on a service level indicated by the user, wherein the service level includes a data access speed.
13 . The system of claim 10 , wherein the storage engine is configured to maintain a metadata tree, wherein for each write request, the metadata tree includes a corresponding entry wherein a key of the entry comprises the user address and a data length, and wherein a value of the entry comprises the one or more corresponding physical addresses on the selected set of M storage drives and the RAID encoding.
14 . A method for RAID storage of data, the method comprising:
in a data storage system having a plurality of storage drives,
receiving one or more write requests to write data on one or more of the plurality of storage drives; and
for each of one or more of the one or more write requests,
determining a corresponding RAID encoding;
determining a number, M, of the plurality of storage drives required for the corresponding RAID encoding, wherein the number is less than a total number, N, of the plurality of storage drives;
determining expected performance for one or more of the plurality of storage drives;
selecting a set of storage drives including M of the plurality of storage drives based at least in part on the expected performance for the plurality of storage drives
writing corresponding data to the selected set of M storage drives using the corresponding RAID encoding.
15 . The method of claim 14 , wherein
determining the expected performance comprises determining a queue depth for each of the N storage drives; and
selecting the set of M storage drives comprises excluding one or more of the storage drives which have a lowest expected performance of the N storage drives and selecting the set of M storage drives from a remainder of the storage drives.
16 . The method of claim 15 , wherein the queue depth for each of the plurality of storage drives comprises a sum of a weighted number of pending reads and a weighted number of pending writes.
17 . The method of claim 15 , wherein a first one of the one or more of the plurality of write requests is written to a first set of M storage drives and a second one of the one or more of the plurality of write requests is written to a second set of M storage drives which is different from the first set of M storage drives.
18 . The method of claim 14 , further comprising determining the RAID encoding corresponding to each write request based at least in part on a service level indicated by the user, wherein the service level includes a redundancy level.
19 . The method of claim 14 , further comprising determining the RAID encoding corresponding to each write request based at least in part on a service level indicated by the user, wherein the service level includes a data access speed.
20 . The method of claim 14 , further comprising maintaining a metadata tree, wherein for each write request, the metadata tree includes a corresponding entry wherein a key of the entry comprises the user address and a data length, and wherein a value of the entry comprises the one or more corresponding physical addresses on the selected set of M storage drives and the RAID encoding.