Automated recovery from raid double failure
Occurrence of a RAID double failure in a slice of a RAID protection group (failed slice) renders data stored in the back-end tracks of the failed slice vulnerable to loss. When a RAID double failure is detected, a new slice is added to the RAID protection group. Front-end tracks that map to the good back-end tracks of the failed slice are moved from the back-end tracks of the failed slice to the back-end tracks of the newly added slice. Any front-end tracks that mapped to the bad back-end tracks of the failed slice are made to be write pending and written to corresponding back-end tracks of the newly added slice. Front-end tracks that map to the bad back-end tracks may be made to be write-pending in connection with a host write operation, by reading the front-end tracks from a local backup, or from a remote backup location.
1. A non-transitory tangible computer readable storage medium having stored thereon a computer program for automated recovery from a RAID double failure, the computer program including a set of instructions which, when executed by a computer, cause the computer to perform a method comprising the steps of:
detecting double failure of a first slice of a RAID group, the first slice of the RAID group including a plurality of good back-end tracks and a plurality of bad back-end tracks;
adding a new second slice to the RAID group, the new second slice having a plurality of second back-end tracks;
for each good back-end track of the first slice of the RAID group, relocating front-end tracks mapped to the good back-end track to a respective one of the second back-end tracks of the new second slice of the RAID group;
identifying a set of front-end tracks mapped to the plurality of bad back-end tracks;
making the front-end tracks write pending; and
writing the front-end tracks to a respective plurality of second back-end tracks of the new second slice of the RAID group.
2. The non-transitory tangible computer readable storage medium of claim 1 , wherein making the front-end tracks write pending comprises, for a particular front-end track associated with a write destage operation, merging previously written data of the write destage operation with write data pending destage, and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
3. The non-transitory tangible computer readable storage medium of claim 1 , wherein making the front-end tracks write pending comprises, for a particular front-end track, retrieving front-end track data from a local protection data repository and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
4. The non-transitory tangible computer readable storage medium of claim 3 , wherein the local protection data repository is a snapshot of the front-end track.
5. The non-transitory tangible computer readable storage medium of claim 1 , wherein making the front-end tracks write pending comprises, for a particular front-end track, retrieving front-end track data from a remote data facility, and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
6. The non-transitory tangible computer readable storage medium of claim 1 , wherein multiple front-end tracks map to each of the bad back-end tracks.
7. The non-transitory tangible computer readable storage medium of claim 1 , further comprising marking the first slice of the RAID group as read-only after detecting the double failure of the first slice of the RAID group.
8. The non-transitory tangible computer readable storage medium of claim 1 , further comprising reclaiming the back-end tracks of the first slice of the RAID group for use in other RAID groups.
9. The non-transitory tangible computer readable storage medium of claim 1 , wherein the RAID group is a RAID 5: 4+1, and the double failure of the first slice of the RAID group is a determination that two or more back-end tracks of the first slice are Data Unavailable Tracks (DUTs).
10. The non-transitory tangible computer readable storage medium of claim 1 , wherein the RAID group is a RAID 6: 12+2, and the double failure of the first slice of the RAID group is a determination that three or more back-end tracks of the first slice are Data Unavailable Tracks (DUTs).
11. A method of automated recovery from a RAID double failure, comprising the steps of:
detecting double failure of a first slice of a RAID group, the first slice of the RAID group including a plurality of good back-end tracks and a plurality of bad back-end tracks;
adding a new second slice to the RAID group, the new second slice having a plurality of second back-end tracks;
for each good back-end track of the first slice of the RAID group, relocating front-end tracks mapped to the good back-end track to a respective one of the second back-end tracks of the new second slice of the RAID group;
identifying a set of front-end tracks mapped to the plurality of bad back-end tracks;
making the front-end tracks write pending; and
writing the front-end tracks to a respective plurality of second back-end tracks of the new second slice of the RAID group.
12. The method of claim 11 , wherein making the front-end tracks write pending comprises, for a particular front-end track associated with a write destage operation, merging previously written data of the write destage operation with write data pending destage, and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
13. The method of claim 11 , wherein making the front-end tracks write pending comprises, for a particular front-end track, retrieving front-end track data from a local protection data repository and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
14. The method of claim 13 , wherein the local protection data repository is a snapshot of the front-end track.
15. The method of claim 11 , wherein making the front-end tracks write pending comprises, for a particular front-end track, retrieving front-end track data from a remote data facility, and writing the particular front-end track to one of the second back-end tracks of the new second slice of the RAID group.
16. The method of claim 11 , wherein multiple front-end tracks map to each of the bad back-end tracks.
17. The method of claim 11 , further comprising marking the first slice of the RAID group as read-only after detecting the double failure of the first slice of the RAID group.
18. The method of claim 11 , further comprising reclaiming the back-end tracks of the first slice of the RAID group for use in other RAID groups.
19. The method of claim 11 , wherein the RAID group is a RAID 5: 4+1, and the double failure of the first slice of the RAID group is a determination that two or more back-end tracks of the first slice are Data Unavailable Tracks (DUTs).
20. The method of claim 11 , wherein the RAID group is a RAID 6: 12+2, and the double failure of the first slice of the RAID group is a determination that three or more back-end tracks of the first slice are Data Unavailable Tracks (DUTs).