LAYERED QUEUE BASED COORDINATION OF POTENTIALLY DESTRUCTIVE ACTIONS IN A DISPERSED STORAGE NETWORK MEMORY
Methods for use in a dispersed storage network (DSN) to coordinate potentially harmful maintenance tasks performed on storage units of the DSN. For each type of maintenance task to be performed on the storage units, an ordered list (e.g., a queue) is generated. Each entry of an ordered list corresponds to a particular storage unit. For each ordered list, a first entry is examined to determine whether to initiate execution of the associated task(s). The determination includes identifying a storage unit associated with the first entry and predicting the impact of performing the task including, for example, the impact on a storage set(s) that includes the identified storage unit. When the predicted impact compares favorably to an impact threshold level, the task is initiated and the selected entry is deleted. When the comparison is unfavorable, the selected entry is moved to another location in the ordered list for postponed execution.
1 . A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN) having a plurality of storage units, the method comprises:
generating an ordered list for each task type of a plurality of tasks to be performed on storage units of the plurality of storage units, including generating at least a first ordered list having a first ordered list entry, wherein each ordered list entry of an ordered list is associated with an individual storage unit of the plurality of storage units; and
determining whether to initiate execution of a task associated with the first ordered list entry, the determining including:
identifying a storage unit associated with the first ordered list entry;
predicting an impact of performing the task on the identified storage unit to generate a predicted impact; and
performing a comparison of the predicted impact to an impact threshold level, and (1) when the comparison is unfavorable, determining not to initiate execution of the task, and (2) when the comparison is favorable, indicating to perform the task on the identified storage unit.
2 . The method of claim 1 , wherein determining not to initiate the execution of the task includes:
identifying another position in the first ordered list; and
moving the first ordered list entry to the another position.
3 . The method of claim 1 , wherein indicating to perform the task on the identified storage unit includes:
issuing a task request for reception by the identified storage unit to facilitate execution of the task; and
removing the first ordered list entry from the first ordered list.
4 . The method of claim 1 , wherein predicting an impact of performing the task includes one or more of:
identifying one or more sets of storage units associated with the storage unit;
obtaining availability information regarding additional storage units associated with the one or more sets of storage units; or
estimating a performance and/or storage reliability level should the storage unit be instructed to perform the task.
5 . The method of claim 1 , wherein the plurality of tasks includes at least one of the following task types: updating hardware, rebooting software, restarting a software process, performing an upgrade, installing a software patch, loading a new software revision, performing an off-line test, or prioritizing tasks associated with an online test.
6 . The method of claim 1 , wherein generating the first ordered list includes maintaining a queue for a task type associated first ordered list, wherein the first ordered list entry corresponds to a top entry in the queue.
7 . The method of claim 1 , wherein generating an ordered list for each task type of a plurality of tasks includes generating a plurality of ordered lists, and wherein determining whether to initiate execution of a task associated with the first ordered list entry further includes selecting the first ordered list entry from an ordered list of the plurality of ordered lists.
8 . The method of claim 7 , wherein selecting the first ordered list entry from an ordered list of the plurality of ordered lists is based on at least one of: the order in which respective entries in the plurality of ordered lists were generated, the number of entries in the plurality of ordered lists, a priority level associated with a task type, storage unit availability levels, a request, or a predetermination.
9 . The method of claim 1 , wherein generating an ordered list for each task type of a plurality of tasks includes generating a plurality of ordered lists, the method repeated for each ordered list entry of the plurality of ordered lists.
10 . A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the DSN having a plurality of storage units, the method comprises:
generating an ordered list for at least one maintenance task type corresponding to a maintenance task to be performed on one or more storage units of the plurality of storage units, including generating at least a first ordered list having a first ordered list entry, wherein each ordered list entry of an ordered list is associated with an individual storage unit of the one or more storage units, and wherein execution of the maintenance task by a storage unit results in temporary unavailability of the storage unit; and
determining whether to initiate execution of the maintenance task associated with the first ordered list entry, the determining including:
identifying the storage unit associated with the first ordered list entry;
predicting an impact of performing the maintenance task on the identified storage unit to generate a predicted impact; and
performing a comparison of the predicted impact to an impact threshold level, and (1) when the comparison is unfavorable, determining not to initiate execution of the maintenance task, and (2) when the comparison is favorable, indicating to perform the maintenance task on the identified storage unit.
11 . The method of claim 10 , wherein the maintenance task corresponds to at least one of the following maintenance task types: updating hardware, rebooting software, restarting a software process, performing an upgrade, installing a software patch, loading a new software revision, performing an off-line test, or prioritizing tasks associated with an online test.
12 . The method of claim 10 , wherein determining not to initiate the execution of the maintenance task includes:
identifying another position in the first ordered list; and
moving the first ordered list entry to the another position.
13 . The method of claim 10 , wherein indicating to perform the maintenance task on the identified storage unit further includes:
issuing a task request for reception by the identified storage unit to facilitate execution of the maintenance task; and
removing the first ordered list entry from the first ordered list.
14 . The method of claim 10 , wherein predicting an impact of performing the maintenance task includes one or more of:
identifying one or more sets of storage units associated with the storage unit;
obtaining availability information regarding additional storage units associated with the one or more sets of storage units; or
estimating a performance and/or storage reliability level should the storage unit be instructed to perform the maintenance task.
15 . A computing device of a group of computing devices of a dispersed storage network (DSN) having a plurality of storage units, the computing device comprises:
a network interface;
a local memory; and
a processing module operably coupled to the network interface and the local memory, wherein the processing module operates to:
generate an ordered list for each task type of a plurality of tasks to be performed on storage units of the plurality of storage units, including at least a first ordered list having a first ordered list entry, wherein each ordered list entry of an ordered list is associated with an individual storage unit of the plurality of storage units; and
determine whether to initiate execution of a task associated with the first ordered list entry, including:
identifying a storage unit associated with the first ordered list entry;
predicting an impact of performing the task on the identified storage unit to generate a predicted impact; and
performing a comparison of the predicted impact to an impact threshold level, and (1) when the comparison is unfavorable, determining not to initiate execution of the task, and (2) when the comparison is favorable, indicating to perform the task on the identified storage unit.
16 . The computing device of claim 15 , wherein determining not to initiate the execution of the task includes:
identifying another position in the first ordered list; and
moving the first ordered list entry to the another position.
17 . The computing device of claim 15 , wherein indicating to perform the task on the identified storage unit includes:
issuing a task request, via the network interface, for reception by the identified storage unit to facilitate execution of the task; and
removing the first ordered list entry from the first ordered list.
18 . The computing device of claim 15 , wherein predicting an impact of performing the task includes:
identifying one or more sets of storage units associated with the storage unit;
obtaining availability information regarding additional storage units associated with the one or more sets of storage units; and
estimating a performance and/or storage reliability level should the storage unit be instructed to perform the task.
19 . The computing device of claim 15 , wherein generating the first ordered list includes maintaining a queue for a task type associated first ordered list, wherein the first ordered list entry corresponds to a top entry in the queue.
20 . The computing device of claim 15 , wherein the plurality of tasks includes at least one of the following task types: updating hardware, rebooting software, restarting a software process, performing an upgrade, installing a software patch, loading a new software revision, performing an off-line test, or prioritizing tasks associated with an online test.