Partial task allocation in a dispersed storage network
A processing system in a dispersed storage and a task (DST) network operates by receiving data and a corresponding task; identifying candidate DST execution units for executing partial tasks of the corresponding task; receiving distributed computing capabilities of the candidate DST execution units; selecting a subset of DST execution units of the candidate DST execution units to favorably execute the partial tasks of the corresponding task; determining task partitioning of the corresponding task into the partial tasks based on one or more of the distributed computing capabilities of the subset of DST execution units; determining processing parameters of the data based on the task partitioning; partitioning the tasks based on the task partitioning to produce the partial tasks; processing the data in accordance with the processing parameters to produce slice groupings; and sending the slice groupings and the partial tasks to the subset of DST execution units.
1. A method for execution by a processing system of a dispersed storage and task (DST) processing unit that includes a processor, the method comprises:
receiving data and a corresponding task;
identifying candidate DST execution units for executing partial tasks of the corresponding task;
receiving computing capabilities of the candidate DST execution units;
selecting a subset of DST execution units of the candidate DST execution units to favorably execute the partial tasks of the corresponding task;
determining task partitioning of the corresponding task into the partial tasks based on one or more of the computing capabilities of the subset of DST execution units and processing parameters of the data corresponding to the task partitioning;
partitioning the tasks based on the task partitioning to produce the partial tasks;
processing the data in accordance with the processing parameters to produce slice groupings; and
sending the slice groupings and the partial tasks to the subset of DST execution units.
2. The method of claim 1 , wherein the computing capabilities are received by one or more of: a query, a lookup, or a message.
3. The method of claim 1 , wherein selecting the subset of DST execution units includes selecting the candidate DST execution units associated with favorable distributed computing capabilities.
4. The method of claim 1 , wherein the computing capabilities include one or more of: a processing capability level, a memory capacity level, a network access level, a bandwidth capability level, an availability level, or a reliability level.
5. The method of claim 1 , wherein selecting the subset of DST execution units includes identifying a number of simultaneous compute resources to execute the task in a favorable timeframe based on the computing capabilities of the candidate DST execution units.
6. The method of claim 1 , wherein determining the task partitioning of the corresponding task into the partial tasks is further based on the processing parameters, and an estimated next data processing destination.
7. The method of claim 1 , wherein determining the task partitioning of the corresponding task into the partial tasks includes aligning the corresponding task with other current tasks and potential future tasks.
8. A processing system of a dispersed storage and task (DST) processing unit comprises:
at least one processor;
a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to perform operations including:
receiving data and a corresponding task;
identifying candidate DST execution units for executing partial tasks of the corresponding task;
receiving computing capabilities of the candidate DST execution units;
selecting a subset of DST execution units of the candidate DST execution units to favorably execute the partial tasks of the corresponding task;
determining task partitioning of the corresponding task into the partial tasks based on one or more of the computing capabilities of the subset of DST execution units and processing parameters of the data corresponding to the task partitioning;
partitioning the tasks based on the task partitioning to produce the partial tasks;
processing the data in accordance with the processing parameters to produce slice groupings; and
sending the slice groupings and the partial tasks to the subset of DST execution units.
9. The processing system of claim 8 , wherein the computing capabilities are received by one or more of: a query, a lookup, or a message.
10. The processing system of claim 8 , wherein selecting the subset of DST execution units includes selecting the candidate DST execution units associated with favorable distributed computing capabilities.
11. The processing system of claim 8 , wherein the computing capabilities include one or more of: a processing capability level, a memory capacity level, a network access level, a bandwidth capability level, an availability level, or a reliability level.
12. The processing system of claim 8 , wherein selecting the subset of DST execution units includes identifying a number of simultaneous compute resources to execute the task in a favorable timeframe based on the computing capabilities of the candidate DST execution units.
13. The processing system of claim 8 , wherein determining the task partitioning of the corresponding task into the partial tasks is further based on the processing parameters, and an estimated next data processing destination.
14. The processing system of claim 8 , wherein determining the task partitioning of the corresponding task into the partial tasks includes aligning the corresponding task with other current tasks and potential future tasks.
15. A non-transitory computer readable storage medium comprises:
at least one memory section that stores operational instructions that, when executed by a processing system of a dispersed storage and task (DST) network that includes a processor and a memory, causes the processing system to perform operations including:
receiving data and a corresponding task;
identifying candidate DST execution units for executing partial tasks of the corresponding task;
receiving computing capabilities of the candidate DST execution units;
selecting a subset of DST execution units of the candidate DST execution units to favorably execute the partial tasks of the corresponding task;
determining task partitioning of the corresponding task into the partial tasks based on one or more of the computing capabilities of the subset of DST execution units and processing parameters of the data corresponding to the task partitioning;
partitioning the tasks based on the task partitioning to produce the partial tasks;
processing the data in accordance with the processing parameters to produce slice groupings; and
sending the slice groupings and the partial tasks to the subset of DST execution units.
16. The non-transitory computer readable storage medium of claim 15 , wherein selecting the subset of DST execution units includes selecting the candidate DST execution units associated with favorable distributed computing capabilities.
17. The non-transitory computer readable storage medium of claim 15 , wherein the computing capabilities include one or more of: a processing capability level, a memory capacity level, a network access level, a bandwidth capability level, an availability level, or a reliability level.
18. The non-transitory computer readable storage medium of claim 15 , wherein selecting the subset of DST execution units includes identifying a number of simultaneous compute resources to execute the task in a favorable timeframe based on the computing capabilities of the candidate DST execution units.
19. The non-transitory computer readable storage medium of claim 15 , wherein determining the task partitioning of the corresponding task into the partial tasks is further based on the processing parameters, and an estimated next data processing destination.
20. The non-transitory computer readable storage medium of claim 15 , wherein determining the task partitioning of the corresponding task into the partial tasks includes aligning the corresponding task with other current tasks and potential future tasks.