Limiting blast radius for critical on-disk data corruptions in a distributed storage system
System and method for managing distributed storage objects for host unavailability in a distributed storage system uses at least one of a crash indicator in a specific on-disk block and a paused object indicator for a distributed storage object to determine whether to perform data recovery for the distributed storage object. When the crash indicator is set or the paused object indicator implies that the distributed storage object is a paused object, the distributed storage object is left as a paused object without perform the data recovery for the distributed storage object. When the crash indicator is unset and the paused object indicator implies that the distributed storage object is not a paused object, the data recovery for the distributed storage object is performed.
1. A computer-implemented method for managing distributed storage objects for host unavailability in a distributed storage system, the method comprising:
checking a crash indicator in a specific on-disk block for a distributed storage object to determine whether the crash indicator is set;
when the crash indicator is set, setting a paused object indicator for the distributed storage object and leaving the distributed storage object as a paused object without performing data recovery for the distributed storage object;
when the crash indicator is unset, checking the paused object indicator for the distributed storage object to determine whether the distributed storage object is a paused object;
when the paused object indicator implies that the distributed storage object is a paused object, leaving the distributed storage object as a paused object without performing the data recovery for the distributed storage object; and
when the paused object indicator implies that the distributed storage object is not a paused object, performing the data recovery for the distributed storage object.
2. The computer-implemented method of claim 1 , wherein checking the crash indicator in the specific on-disk block for the distributed storage object includes checking a single crash bit in specific on-disk block for the distributed storage object to determine whether the single crash bit is set.
3. The computer-implemented method of claim 1 , wherein checking the paused object indicator for the distributed storage object includes checking at least one of an on-disk paused object indicator and a distributed consensus service to determine whether the distributed storage object is a paused object.
4. The computer-implemented method of claim 3 , wherein the on-disk paused object indicator is a single bit in the specific on-disk block for the distributed storage object.
5. The computer-implemented method of claim 3 , wherein checking the paused object indicator for the distributed storage object further includes, when one of the on-disk paused object indicator and the distributed consensus service indicates that the distributed storage object is a paused object, setting the other one of the on-disk paused object indicator and the distributed consensus service to indicate that the distributed storage object is a paused object and leaving the distributed storage object as a paused object without performing the data recovery for the distributed storage object.
6. The computer-implemented method of claim 3 , wherein checking the paused object indicator for the distributed storage object further includes, when both of the on-disk paused object indicator and the distributed consensus service indicate that the distributed storage object is not a paused object, performing the data recovery for the distributed storage object.
7. The computer-implemented method of claim 3 , further comprising, when the distributed consensus service indicate that the distributed storage object is to be resumed, unsetting the on-disk paused object indicator to indicate that the distributed storage object is not a paused object and performing the data recovery for the distributed storage object.
8. The computer-implemented method of claim 1 , wherein performing the data recovery for the distributed storage object includes setting the crash indicator when the data recovery is started and unsetting the crash indicator when the data recover is completed.
9. The computer-implemented method of claim 1 , further comprising electing a new leader host for the distributed storage object in response to unavailability of a prior leader host for the distributed storage object prior to checking the crash indicator.
10. A non-transitory computer-readable storage medium containing program instructions for managing distributed storage objects for host unavailability in a distributed storage system, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:
checking a crash indicator in a specific on-disk block for a distributed storage object to determine whether the crash indicator is set;
when the crash indicator is set, setting a paused object indicator for the distributed storage object and leaving the distributed storage object as a paused object without performing data recovery for the distributed storage object;
when the crash indicator is unset, checking the paused object indicator for the distributed storage object to determine whether the distributed storage object is a paused object;
when the paused object indicator implies that the distributed storage object is a paused object, leaving the distributed storage object as a paused object without performing the data recovery for the distributed storage object; and
when the paused object indicator implies that the distributed storage object is not a paused object, performing the data recovery for the distributed storage object.
11. The computer-readable storage medium of claim 10 , wherein checking the crash indicator in the specific on-disk block for the distributed storage object includes checking a single crash bit in specific on-disk block for the distributed storage object to determine whether the single crash bit is set.
12. The computer-readable storage medium of claim 10 , wherein checking the paused object indicator for the distributed storage object includes checking at least one of an on-disk paused object indicator and a distributed consensus service to determine whether the distributed storage object is a paused object.
13. The computer-readable storage medium of claim 12 , wherein the on-disk paused object indicator is a single bit in the specific on-disk block for the distributed storage object.
14. The computer-readable storage medium of claim 12 , wherein checking the paused object indicator for the distributed storage object further includes, when one of the on-disk paused object indicator and the distributed consensus service indicates that the distributed storage object is a paused object, setting the other one of the on-disk paused object indicator and the distributed consensus service to indicate that the distributed storage object is a paused object and leaving the distributed storage object as a paused object without performing the data recovery for the distributed storage object.
15. The computer-readable storage medium of claim 12 , wherein checking the paused object indicator for the distributed storage object further includes, when both of the on-disk paused object indicator and the distributed consensus service indicate that the distributed storage object is not a paused object, performing the data recovery for the distributed storage object.
16. The computer-readable storage medium of claim 12 , further comprising, when the distributed consensus service indicate that the distributed storage object is to be resumed, unsetting the on-disk paused object indicator to indicate that the distributed storage object is not a paused object and performing the data recovery for the distributed storage object.
17. The computer-readable storage medium of claim 10 , wherein performing the data recovery for the distributed storage object includes setting the crash indicator when the data recovery is started and unsetting the crash indicator when the data recover is completed.
18. The computer-readable storage medium of claim 10 , wherein the steps further comprise electing a new leader host for the distributed storage object in response to unavailability of a prior leader host for the distributed storage object prior to checking the crash indicator.
19. A computer system comprising:
memory; and
at least one processor configured to:
check a crash indicator in a specific on-disk block for a distributed storage object to determine whether the crash indicator is set;
when the crash indicator is set, set a paused object indicator for the distributed storage object and leave the distributed storage object as a paused object without performing data recovery for the distributed storage object;
when the crash indicator is unset, check the paused object indicator for the distributed storage object to determine whether the distributed storage object is a paused object;
when the paused object indicator implies that the distributed storage object is a paused object, leave the distributed storage object as a paused object without performing the data recovery for the distributed storage object; and
when the paused object indicator implies that the distributed storage object is not a paused object, perform the data recovery for the distributed storage object.
20. The computer system of claim 19 , wherein the at least one processor is configured to check at least one of an on-disk paused object indicator and a distributed consensus service to determine whether the distributed storage object is a paused object.