MULTI-LAYER DISTRIBUTION OF A COMPUTING TASK IN A DISPERSED STORAGE NETWORK
Methods for use in a dispersed storage network (DSN) to determine distribution of computing tasks. A computing device receives a partial task and associated contiguous data and determines whether to process the partial task locally. When processing locally, the computing device determines execution steps and a schedule, identifies a portion of the contiguous data, and executes the execution steps, in accordance with the schedule, on the portion of data to produce a partial result. When not processing the partial task locally, the computing device selects a portion of the contiguous data and determines processing parameters based. The computing device further determines task partitioning to transform the partial task into one or more secondary partial tasks, processes the select data in accordance with the processing parameters to produce secondary data, and sends the secondary data and one or more corresponding secondary partial tasks to storage units of the DSN.
1 . A method for execution by one or more processing modules of a computing device of a dispersed storage network (DSN), the method comprises:
receiving at least one partial task associated with a group of slices of contiguous data;
receiving the group of slices of contiguous data;
determining whether to process the at least one partial task locally;
when determining to process the at least one partial task locally:
determining execution steps and a schedule;
identifying a portion of the contiguous data for execution of one or more steps of the execution steps; and
executing the one or more steps of the execution steps, in accordance with the schedule, on the portion of the contiguous data to produce a partial result;
when determining not to process the at least one partial task locally:
selecting a portion of the contiguous data as select data;
determining processing parameters of the select data based, at least in part, on a number of storage units;
determining task partitioning, based on the number of storage units and the processing parameters, to transform the at least one partial task into one or more secondary partial tasks;
processing the select data in accordance with the processing parameters to produce secondary slice groupings; and
sending the secondary slice groupings and one or more corresponding secondary partial tasks to storage units of the DSN.
2 . The method of claim 1 further comprises:
when determining not to process the at least one partial task locally:
receiving, from the storage units, one or more secondary partial results; and
processing the one or more secondary partial results to produce a partial result for the at least one partial task.
3 . The method of claim 2 , wherein processing the one or more secondary partial results includes at least one of decoding the one or more secondary partial results or aggregating the one or more secondary partial results.
4 . The method of claim 3 further comprises sending the partial result to a requesting entity.
5 . The method of claim 3 further comprises facilitating storage of the partial result in the DSN.
6 . The method of claim 1 , wherein determining whether to process the at least one partial task locally is based on one or more of a local task execution capacity level, a required task execution capacity level, or a comparison of the difference of the local task execution capacity level and the required task execution capacity level to a difference threshold.
7 . The method of claim 6 further comprises:
determining to process the at least one partial task locally when the difference of the local task execution capacity level and the required task execution capacity level compares favorably to the difference threshold.
8 . The method of claim 1 , wherein determining whether to process the at least one partial task locally is based on one or more of comparing an amount of data of the group of slices of contiguous data to a data threshold, a partial task type, task execution resource availability, or a task schedule.
9 . A computing device for use in a dispersed storage network (DSN), the computing device comprises:
a network interface;
a local memory comprising instructions; and
a processing module operably coupled to the network interface and the local memory, wherein the processing module executes the instructions to:
receive, via the network interface, at least one partial task associated with a group of slices of contiguous data;
receive the group of slices of contiguous data;
determine whether to process the at least one partial task locally;
when determining to process the at least one partial task locally:
determine execution steps and a schedule;
identify a portion of the contiguous data for execution of one or more steps of the execution steps; and
execute the one or more steps of the execution steps, in accordance with the schedule, on the portion of the contiguous data to produce a partial result;
when determining not to process the at least one partial task locally:
select a portion of the contiguous data as select data;
determine processing parameters of the select data based, at least in part, on a number of storage units;
determine task partitioning, based on the number of storage units and the processing parameters, to transform the at least one partial task into one or more secondary partial tasks;
process the select data in accordance with the processing parameters to produce secondary slice groupings; and
send, via the network interface, the secondary slice groupings and one or more corresponding secondary partial tasks to storage units of the DSN.
10 . The computing device of claim 9 , wherein the processing module further executes the instructions to:
when determining not to process the at least one partial task locally:
receive, via the network interface, one or more secondary partial results; and
process the one or more secondary partial results to produce a partial result for the at least one partial task.
11 . The computing device of claim 10 , wherein processing the one or more secondary partial results includes at least one of decoding the one or more secondary partial results or aggregating the one or more secondary partial results.
12 . The computing device of claim 11 , wherein the processing module further executes the instructions to:
send, via the network interface, the partial result to a requesting entity.
13 . The computing device of claim 11 , wherein the processing module further executes the instructions to:
facilitate storage of the partial result in the DSN.
14 . The computing device of claim 9 , wherein determining whether to process the at least one partial task locally is based on one or more of a local task execution capacity level, a required task execution capacity level, or a comparison of the difference of the local task execution capacity level and the required task execution capacity level to a difference threshold.
15 . The computing device of claim 14 , wherein the processing module further executes the instructions to:
determine to process the at least one partial task locally when the difference of the local task execution capacity level and the required task execution capacity level compares favorably to the difference threshold.
16 . The computing device of claim 9 , wherein determining whether to process the at least one partial task locally is based on one or more of comparing an amount of data of the group of slices of contiguous data to a data threshold, a partial task type, task execution resource availability, or a task schedule.
17 . A computer readable storage medium having operational instructions embodied therewith, the operational instructions executable by one or more processing modules of a dispersed storage network (DSN) to cause the one or more processing modules to:
receive at least one partial task associated with a group of slices of contiguous data;
receive the group of slices of contiguous data;
determine whether to process the at least one partial task locally;
when determining to process the at least one partial task locally:
determine execution steps and a schedule;
identify a portion of the contiguous data for execution of one or more steps of the execution steps; and
execute the one or more steps of the execution steps, in accordance with the schedule, on the portion of the contiguous data to produce a partial result;
when determining not to process the at least one partial task locally:
select a portion of the contiguous data as select data;
determine processing parameters of the select data based, at least in part, on a number of storage units;
determine task partitioning, based on the number of storage units and the processing parameters, to transform the at least one partial task into one or more secondary partial tasks;
process the select data in accordance with the processing parameters to produce secondary slice groupings; and
send the secondary slice groupings and one or more corresponding secondary partial tasks to storage units of the DSN.
18 . The computer readable storage medium of claim 17 , wherein the operational instructions are further executable to cause the one or more processing modules to:
when determining not to process the at least one partial task locally:
receive, from the storage units, one or more secondary partial results; and
process the one or more secondary partial results to produce a partial result for the at least one partial task.
19 . The computer readable storage medium of claim 18 , wherein processing the one or more secondary partial results includes at least one of decoding the one or more secondary partial results or aggregating the one or more secondary partial results.
20 . The computer readable storage medium of claim 19 , wherein the operational instructions are further executable to cause the one or more processing modules to:
send the partial result to a requesting entity.