Dynamic raid expansion
Dynamic RAID expansion is disclosed, including: incrementally expanding a redundant storage system comprising a plurality of storage drives by adding one or more storage drives to the plurality of storage drives; and storing, for each stripe included in the redundant storage system, metadata indicating which storage drives in the redundant storage system store valid data for the stripe.
1. A redundant storage system comprising a plurality of storage drives, comprising:
a processor configured to:
receive an indication to expand the redundant storage system, wherein the plurality of storage drives includes an original number of storage drives, wherein the redundant storage system comprises at least a first stripe and a second stripe;
expand the redundant storage system by adding one or more new storage drives to the plurality of storage drives, wherein a combination of the one or more new storage drives and the plurality of storage drives includes an expanded number of storage drives, wherein the expanded number of storage drives is greater than the original number of storage drives;
write first valid data across the expanded number of storage drives in the first stripe associated with the redundant storage system; and
store metadata that indicates that:
the first stripe stores the first valid data across the expanded number of storage drives; and
the second stripe stores second valid data across the original number of storage drives but not in the one or more new storage drives.
2. The redundant storage system of claim 1 , wherein the redundant storage system comprises a redundant array of independent disks (RAID) system.
3. The redundant storage system of claim 1 , wherein to write the first valid data across the expanded number of storage drives in the first stripe associated with the redundant storage system comprises to:
detect that the one or more new storage drives are added to the plurality of storage drives;
expand a size of a buffer to span the first stripe across the expanded number of storage drives;
store a plurality of objects in the buffer;
determine that the buffer has met a flush condition; and
write the plurality of objects stored in the buffer to the first stripe.
4. The redundant storage system of claim 3 , wherein the buffer comprises an in-memory buffer.
5. The redundant storage system of claim 3 , wherein the flush condition comprises at least a threshold percentage of a capacity of the buffer being filled with data.
6. The redundant storage system of claim 1 , wherein the processor is further configured to:
receive an object read request;
obtain a stripe number, a logical drive number, and an offset within a stripe unit from the object read request;
retrieve the metadata; and
use the metadata, the stripe number, the logical drive number, and the offset within the stripe unit to process the object read request.
7. The redundant storage system of claim 1 , wherein the processor is further configured to:
read stripe unit headers corresponding to respective ones of at least a subset of stripe units associated with each stripe included in a plurality of stripes of the redundant storage system to determine which storage drives store valid data in that stripe;
generate the metadata based at least in part on the read stripe unit headers; and
store the metadata.
8. The redundant storage system of claim 1 , wherein the processor is further configured to:
receive an object read request;
read stripe unit headers associated with a third stripe that is identified in the object read request to determine which one or more storage drives store valid data in the third stripe; and
update the metadata based on the read stripe unit headers.
9. The redundant storage system of claim 1 , wherein the processor is further configured to:
determine that a third stripe included in the redundant storage system has been written to; and
update the metadata based at least in part on the third stripe included in the redundant storage system that has been written to.
10. The redundant storage system of claim 1 , wherein the processor is further configured to:
determine that a third stripe included in the redundant storage system has been written to;
update log data to include which stripe units within the third stripe now store valid data; and
update the metadata based at least in part on the log data.
11. The redundant storage system of claim 1 , wherein the processor is further configured to:
receive an object read request;
obtain a stripe number, a logical drive number, an offset within a stripe unit, and a number of storage drives in the redundant storage system that stores valid data for a stripe associated with the stripe number from the object read request; and
use the stripe number, the logical drive number, the offset within the stripe unit, and the number of storage drives in the redundant storage system that stores valid data for the stripe associated with the stripe number to process the object read request.
12. The redundant storage system of claim 1 , wherein the redundant storage system is permitted to be expanded by N number of storage drives in one incremental expansion, where N is an integer that is at least one.
13. The redundant storage system of claim 1 , wherein the processor is further configured to:
detect that the one or more new storage drives are added to the redundant storage system;
write to each of the plurality of storage drives and the one or more new storage drives of the redundant storage system a new group header that indicates that the plurality of storage drives and the one or more new storage drives are part of the redundant storage system;
determine that fewer than each one of the plurality of storage drives and the one or more new storage drives is detected by a new controller; and
cause the new controller to use a previous group header that indicates that the redundant storage system comprises the plurality of storage drives but not the one or more new storage drives.
14. The redundant storage system of claim 1 , wherein the processor is further configured to:
detect that the one or more new storage drives are added to the redundant storage system;
write to each of the plurality of storage drives and the one or more new storage drives a new group header that indicates that the plurality of storage drives and the one or more new storage drives are part of the redundant storage system;
determine that no fewer than each one of the plurality of storage drives and the one or more new storage drives is detected by a new controller; and
cause the new controller to use the new group header that indicates that the redundant storage system comprises the plurality of storage drives and the one or more new storage drives.
15. A method, comprising:
receiving an indication to expand a redundant storage system, wherein the redundant storage system comprises a plurality of storage drives, wherein the plurality of storage drives includes an original number of storage drives, wherein the redundant storage system comprises at least a first stripe and a second stripe;
expanding the redundant storage system by adding one or more new storage drives to the plurality of storage drives, wherein a combination of the one or more new storage drives and the plurality of storage drives includes an expanded number of storage drives, wherein the expanded number of storage drives is greater than the original number of storage drives;
writing first valid data across the expanded number of storage drives in the first stripe associated with the redundant storage system; and
storing metadata that indicates that:
the first stripe stores the first valid data across the expanded number of storage drives; and
the second stripe stores second valid data across the original number of storage drives but not in the one or more new storage drives.
16. The method of claim 15 , further comprising:
receiving an object read request;
obtaining a stripe number, a logical drive number, and an offset within a stripe unit from the object read request;
retrieving the metadata; and
using the metadata, the stripe number, the logical drive number, and the offset within the stripe unit to process the object read request.
17. The method of claim 15 , further comprising:
reading stripe unit headers corresponding to respective ones of at least a subset of stripe units associated with each stripe included in a plurality of stripes of the redundant storage system to determine which storage drives store valid data in that stripe;
generating the metadata based at least in part on the read stripe unit headers; and
storing the metadata.
18. The method of claim 15 , further comprising:
receiving an object read request;
obtaining a stripe number, a logical drive number, an offset within a stripe unit, and a number of storage drives in the redundant storage system that stores valid data for a stripe associated with the stripe number from the object read request; and
using the stripe number, the logical drive number, the offset within the stripe unit, and the number of storage drives in the redundant storage system that stores valid data for the stripe associated with the stripe number to process the object read request.
19. The method of claim 15 , wherein the redundant storage system is permitted to be expanded by N number of storage drives in one incremental expansion, where N is an integer that is at least one.
20. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving an indication to expand a redundant storage system, wherein the redundant storage system comprises a plurality of storage drives, wherein the plurality of storage drives includes an original number of storage drives, wherein the redundant storage system comprises at least a first stripe and a second stripe;
expanding the redundant storage system by adding one or more new storage drives to the plurality of storage drives, wherein a combination of the one or more new storage drives and the plurality of storage drives includes an expanded number of storage drives, wherein the expanded number of storage drives is greater than the original number of storage drives;
writing first valid data across the expanded number of storage drives in the first stripe associated with the redundant storage system; and
storing metadata that indicates that:
the first stripe stores the first valid data across the expanded number of storage drives; and
the second stripe stores second valid data across the original number of storage drives but not in the one or more new storage drives.