Partial task execution in a dispersed storage network
A processing system in a dispersed storage and a task network DSTN determines whether or not to execute a partial task. When determined to execute the partial task, the processing system operates by determining execution steps and a schedule; identifying a portion of the contiguous data for execution of one or more steps of the execution steps; retrieving the portion of the contiguous data from the disk drive; executing the one or more steps of the execution steps in accordance with the schedule on the portion of the contiguous data to produce a partial result; dispersed storage error encoding the partial result to produce a plurality of sets of slices in accordance with dispersal parameters associated with one or more of the group of slices and the partial task; and facilitating storing a plurality of sets of slices in the DSTN.
1. A method for execution by a processing system of a dispersed storage and task (DST) execution unit that includes a processor, the method comprises: receiving a partial task with regards to a group of slices of data; receiving slices of the group of slices to produce received slices; determining when a threshold number of the received slices has been received; when the threshold number of the received slices has been received, sending the received slices to a disk drive for storage; determining whether or not to execute the partial task; when determined to execute the partial task: determining execution steps and a schedule; identifying a portion of the data for execution of one or more steps of the execution steps; retrieving the portion of the data from the disk drive; executing the one or more steps of the execution steps in accordance with the schedule on the portion of the data to produce a partial result; and dispersed storage error encoding the partial result to produce a plurality of sets of slices stored in a dispersed storage and task network (DSTN) in accordance with dispersal parameters associated with one or more of the group of slices and the partial task.
2. The method of claim 1 , wherein the threshold number corresponds to a maximum number of bytes limited by an ingestion cache memory.
3. The method of claim 1 , wherein determining whether or not to execute the partial task is based on one or more of: comparing an amount of data in the received slices to a data threshold, a partial task type corresponding to the partial task, a task execution resource availability or a task schedule.
4. The method of claim 1 , wherein determining whether or not to execute the partial includes determining to execute the partial task when data of the received slices can be processed in accordance with the partial task.
5. The method of claim 1 , wherein retrieving the portion of the data from the disk drive includes receiving a data stream from the disk drive that contains the portion of the data.
6. The method of claim 1 , further comprising:
when the determined not to execute the partial task:
determining whether or not more slices of the group of slices are expected;
when more slices are expected, receiving the more slices of the group of slices to produce more received slices.
7. The method of claim 6 , wherein determining whether or not more slices of the group of slices are expected is based on one or more of: a data size indicator, a query, a lookup, or a number of bytes in the received slices.
8. The method of claim 1 , wherein determining the execution steps and the schedule is based on one or more of: the partial task, the data, a previous task schedule, a schedule template, a task execution resource availability level or a task execution requirement.
9. The method of claim 1 , wherein identifying the portion of the data includes matching the portion of the data to the one or more steps of execution steps based on one or more of: a data type indicator associated with the portion, a data type associated with one or more steps or a data available indicator.
10. The method of claim 1 , wherein retrieving the portion of the data from the disk drive includes accessing the disk drive for multiple data bytes of the portion of the data.
11. The method of claim 1 , wherein facilitating the storing of the plurality of sets of slices includes storing of the plurality of sets of slices with a common pillar number.
12. The method of claim 1 , wherein facilitating the storing of the plurality of sets of slices includes storing of the plurality of sets of slices with two or more pillar numbers.
13. A processing system of a dispersed storage and task (DST) execution unit comprises: at least one processor; a memory that stores operational instructions, that when executed by the at least one processor cause the processing system to perform operations including: receiving a partial task with regards to a group of slices of data; receiving slices of the group of slices to produce received slices; determining when a threshold number of the received slices has been received; when the threshold number of the received slices has been received, sending the received slices to a disk drive for storage; determining whether or not to execute the partial task; when determined to execute the partial task: determining execution steps and a schedule; identifying a portion of the data for execution of one or more steps of the execution steps; retrieving the portion of the data from the disk drive; executing the one or more steps of the execution steps in accordance with the schedule on the portion of the data to produce a partial result; and dispersed storage error encoding the partial result to produce a plurality of sets of slices stored in a dispersed storage and task network (DSTN) in accordance with dispersal parameters associated with one or more of the group of slices and the partial task.
14. The processing system of claim 13 , wherein the threshold number corresponds to a maximum number of bytes limited by an ingestion cache memory.
15. The processing system of claim 13 , wherein determining whether or not to execute the partial task is based on one or more of: comparing an amount of data in the received slices to a data threshold, a partial task type corresponding to the partial task, a task execution resource availability or a task schedule.
16. The processing system of claim 13 , wherein determining whether or not to execute the partial includes determining to execute the partial task when data of the received slices can be processed in accordance with the partial task.
17. The processing system of claim 13 , wherein retrieving the portion of the data from the disk drive includes receiving a data stream from the disk drive that contains the portion of the data.
18. The processing system of claim 13 , further comprising:
when the determined not to execute the partial task:
determining whether or not more slices of the group of slices are expected;
when more slices are expected, receiving the more slices of the group of slices to produce more received slices.
19. The processing system of claim 13 , wherein determining the execution steps and the schedule is based on one or more of: the partial task, the data, a previous task schedule, a schedule template, a task execution resource availability level or a task execution requirement.
20. A computer readable storage medium comprises: at least one memory section including hardware that stores operational instructions that, when executed by a processing system of a dispersed storage network (DSN) that includes a processor and a memory, causes the processing system to perform operations including: receiving a partial task with regards to a group of slices of data; receiving slices of the group of slices to produce received slices; determining when a threshold number of the received slices has been received; when the threshold number of the received slices has been received, sending the received slices to a disk drive for storage; determining whether or not to execute the partial task; when determined to execute the partial task: determining execution steps and a schedule; identifying a portion of the data for execution of one or more steps of the execution steps; retrieving the portion of the data from the disk drive; executing the one or more steps of the execution steps in accordance with the schedule on the portion of the data to produce a partial result; and dispersed storage error encoding the partial result to produce a plurality of sets of slices stored in a dispersed storage and task network (DSTN) in accordance with dispersal parameters associated with one or more of the group of slices and the partial task.