Storage system spanning multiple failure domains
A plurality of failure domains are communicatively coupled to each other via a network, and each of the plurality of failure domains is coupled to one or more storage devices. A failure resilient stripe is distributed across the plurality of storage devices, such that two or more blocks of the failure resilient stripe are located in each failure domain.
1. A system comprising:
a first failure domain comprising a first backend and a first storage device; and
a second failure domain comprising a second backend and a second storage device, wherein:
the first backend and the second backend reach consensus that the first backend is an active controller,
the first backend is operable to build a failure resilient stripe comprising a plurality of blocks,
two or more blocks of the plurality of blocks comprise error correction information,
two or more blocks of the plurality of blocks are located in the first storage device,
two or more blocks of the plurality of blocks are located in the second storage device,
if a data access is directed to the second backend, the second backend redirects the data access to the first backend,
upon a failure of the first failure domain, the two or more blocks of the plurality of blocks in the first storage device are regenerated according to the two or more blocks of the plurality of blocks in the second storage device, and
upon a failure of the second failure domain, the two or more blocks of the plurality of blocks in the second storage device are regenerated according to the two or more blocks of the plurality of blocks in the first storage device.
2. The system of claim 1 , wherein the first failure domain comprises a plurality of solid-state drives.
3. The system of claim 1 , wherein the first failure domain comprises one or more servers.
4. The system of claim 1 , wherein the first failure domain comprises a rack and a network switch.
5. The system of claim 1 , wherein if the first failure domain and the second failure domain lose communication with each other, a third failure domain is operable to determine which of the first failure domain and the second failure domain will continue running the system by rebuilding data.
6. The system of claim 1 , wherein neither the first failure domain nor the second failure domain will rebuild the failure resilient stripe unless permission is granted by a third failure domain.
7. The system of claim 1 , wherein the first backend comprises a bucket that initially builds the failure resilient stripe.
8. The system of claim 1 , wherein the first backend comprises a bucket that is a leader of the failure resilient stripe.
9. The system of claim 1 , wherein the second backend comprises a bucket that becomes a leader of the failure resilient stripe if the first failure domain fails.
10. The system of claim 1 , wherein the first failure domain comprise an availability group.
11. A method comprising:
receiving a plurality of data pieces by a first failure domain;
generating a plurality of error correction pieces according to the plurality of data pieces;
building a failure resilient stripe comprising a plurality of blocks using a first backend of the first failure domain, wherein each block of the plurality of blocks comprises one data piece of the plurality of data pieces or one error correction piece of the plurality of error correction pieces;
placing a first two or more blocks of the plurality of blocks in the first failure domain;
placing a second two or more blocks of the plurality of blocks in a second failure domain;
the first backend and a second backend of the second failure domain reach consensus that the first backend is an active controller;
if a data access is directed to the second backend, the second backend redirects the data access to the first backend;
if the first failure domain fails, regenerating the first two or more blocks in the first failure domain according to the second two or more blocks in the second failure domain; and
if the second failure domain fails, regenerating the second two or more blocks in the second failure domain according to the first two or more blocks in the first failure domain.
12. The method of claim 11 , wherein the first failure domain comprises a plurality of storage devices.
13. The method of claim 11 , wherein the first failure domain comprises one or more servers.
14. The method of claim 11 , wherein the first failure domain comprises a rack and a network switch.
15. The method of claim 11 , wherein the method comprises determining which of the first failure domain and the second failure domain will rebuild the failure resilient stripe if the first failure domain and the second failure domain lose communication with each other.
16. The method of claim 11 , wherein neither the first failure domain nor the second failure domain will rebuild the failure resilient stripe unless permission is granted by a third failure domain.
17. The method of claim 11 , wherein the failure resilient stripe is built by a bucket in the first backend.
18. The method of claim 11 , wherein a bucket of the first backend is a leader of the failure resilient stripe.
19. The method of claim 11 , wherein the method comprises promoting a bucket of the second backend of the second failure domain to become a leader of the failure resilient stripe if the first failure domain fails.
20. The method of claim 11 , wherein the first failure domain comprise an availability group.