Automated stalled process detection and recovery
A dispersed storage network (DSN) includes multiple storage units. A processing unit included in the DSN issues an access request to one of the storage units, and identifies the storage unit as a failing storage unit based, at least in part, on a rate of growth of a network queue associated with the storage unit. the processing unit then issues an error indicator to a recovery unit for further action.
1. A method for use in a dispersed storage network (DSN) including a plurality of storage units, the method comprising:
issuing an access request from a processing unit included in the DSN to a storage unit of the plurality of storage units;
identifying, by the processing unit, the storage unit as a failing storage unit based, at least in part, on a rate of growth of a network queue associated with the storage unit; and
issuing, by the processing unit, an error indicator to a recovery unit for further action.
2. The method of claim 1 , further comprising:
determining, by the processing unit, that the failing storage unit is not responding to at least some access requests.
3. The method of claim 1 , wherein the error indicator includes:
information identifying the failing storage unit and a failure type.
4. The method of claim 1 , further comprising:
issuing, by the recovery unit, a test request to the failing storage unit; and
determining, by the recovery unit, to implement a corrective action based on a result of the test request.
5. The method of claim 4 , wherein issuing the test request includes:
generating a particular test request based on a failure type associated with the failing storage unit; and
sending the particular test request to the failing storage unit.
6. The method of claim 4 , wherein determining to implement the corrective action includes:
determining at least one of receiving an unfavorable response to the test request, or determining that a time period has expired without receiving a test response.
7. The method of claim 1 , further comprising:
issuing a corrective action command, by the recovery unit, to a managing unit associated with the failing storage unit.
8. The method of claim 1 , wherein the error indicator includes:
information identifying the failing storage unit as partially non-responsive.
9. A dispersed storage network (DSN) including a plurality of storage units, the DSN comprising:
a processing unit configured to:
issue an access request to a storage unit of the plurality of storage units;
identify the storage unit as a failing storage unit based, at least in part, on a rate of growth of a network queue associated with the storage unit;
issue an error indicator to a recovery unit, the error indicator indicate the failing storage unit; and
the recovery unit configured to implement corrective action in accordance with the error indication.
10. The dispersed storage network (DSN) of claim 9 , wherein the processing unit is configured to:
determine that the failing storage unit is not responding to at least some access requests.
11. The dispersed storage network (DSN) of claim 9 , wherein the error indicator includes:
information identifying the failing storage unit and a failure type.
12. The dispersed storage network (DSN) of claim 9 , wherein the recovery unit is further configured to:
issue a test request to the failing storage unit; and
determine to implement the corrective action based on a result of the test request.
13. The dispersed storage network (DSN) of claim 12 , wherein the recovery unit is configured to issue the test request, at least in part, by:
generating a particular test request based on a failure type associated with the failing storage unit; and
sending the particular test request to the failing storage unit.
14. The dispersed storage network (DSN)of claim 12 , wherein the recovery unit is configured to issue implement the corrective action, at least in part, by:
determining at least one of receiving an unfavorable response to the test request, or determining that a time period has expired without receiving a test response.
15. The dispersed storage network (DSN)of claim 9 , wherein the recovery unit is configured to:
issue a corrective action command to a managing unit associated with the failing storage unit.
16. The dispersed storage network (DSN) of claim 9 , wherein the error indicator includes:
information identifying the failing storage unit as partially non-responsive.
17. A processing unit included in a dispersed storage network (DSN) including a plurality of storage units, the processing unit comprising:
a processor;
a memory coupled to the processor;
a non-transitory program of instructions configured to be stored in the memory and executed by the processor, the program of instructions including:
at least one instruction to issue an access request to a storage unit of the plurality of storage units;
at least one instruction to identify, by the processing unit, the storage unit as a failing storage unit based, at least in part, on a rate of growth of a network queue associated with the storage unit; and
at least one instruction to issue, by the processing unit, an error indicator to a recovery unit for further action.
18. The recovery unit of claim 17 , wherein the at least one instruction to identify includes:
determine that the failing storage unit is not responding to at least some access requests.
19. The recovery unit of claim 17 , wherein the error indicator includes:
information identifying the failing storage unit and a failure type.
20. The recovery unit of claim 19 , wherein the error indicator includes:
information identifying the failing storage unit as partially non-responsive.