Distributed storage and computing of interim data
View Patent ↗A method begins by a set of distributed storage and task (DST) execution units receiving a set of partial tasks and data, where a partial task of the set of partial tasks includes a common task and a unique partial sub-task. The method continues with the set of DST execution units executing the common task on the data to produce a set of preliminary partial results. The method continues with a first DST execution unit of the set of DST execution units generating first interim data based on the at least some of the set of preliminary partial results. The method continues with the first DST execution unit executing a first unique partial sub-task on at least one of a first portion of the data and the first interim data to produce a first partial result.
1. A computing device comprising:
an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);
memory that stores operational instructions; and
a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to:
receive, via the interface and from a first other computing device, a first partial task of a set of partial tasks and a first portion of data, wherein the first partial task of the set of partial tasks includes a common task and a first unique partial sub-task;
execute the common task on the first portion of the data to produce a first preliminary partial result of a set of preliminary partial results;
generate first interim data based on at least some of the set of preliminary partial results;
execute the first unique partial sub-task on at least one of the first portion of the data or the first interim data to produce a first partial result;
generate first partial redundancy data based on the first interim data;
transmit, via the interface and to a second other computing device, the first partial redundancy data to be processed based on decoding by the second other computing device in accordance with a dispersed storage error coding function to produce redundancy data; and
transmit, via the interface and to at least one of the second other computing device or a third other computing device, the first partial result to be processed by the at least one of the second other computing device or the third other computing device to produce a result corresponding to at least one of the set of partial tasks performed on the data.
2. The computing device of claim 1 , wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:
execute the common task on the first portion of the data in cooperation with at least one other computing device executing the common task on at least one other portion of the data to produce the set of preliminary partial results, wherein the at least one other computing device producing at least one other preliminary partial result of the set of preliminary partial results.
3. The computing device of claim 2 , wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:
generate the first interim data based on the first preliminary partial result of the set of preliminary partial results and one or more of the at least one other preliminary partial result of the set of preliminary partial results.
4. The computing device of claim 1 , wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:
receive the first partial task of the set of partial tasks and the first portion of the data in cooperation with at least one other computing device configured to receive a second partial task of the set of partial tasks and a second portion of the data.
5. The computing device of claim 4 , wherein the data is allocated into the first portion of the data and the second portion of the data based on at least one of a time parameter, a geographic parameter, or a source parameter.
6. The computing device of claim 1 , wherein:
the common task corresponding to a function to identify a common characteristic of portions of the data including the first portion of the data; and
the first unique partial sub-task corresponding to a function to categorize uniquely the common characteristic of the portions of the data based on a unique parameter.
7. The computing device of claim 1 further comprising:
a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.
8. The computing device of claim 1 , wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).
9. A computing device comprising:
an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);
memory that stores operational instructions; and
a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to:
receive, via the interface and from a first other computing device, a first partial task of a set of partial tasks and a first portion of data in cooperation with at least one other computing device configured to receive a second partial task of the set of partial tasks and a second portion of the data, wherein the first partial task of the set of partial tasks includes a common task and a first unique partial sub-task, and wherein the second partial task of the set of partial tasks includes the common task and a second unique partial sub-task;
execute the common task on the first portion of the data to produce a first preliminary partial result of a set of preliminary partial results in cooperation with at least one other computing device executing the common task on at least one other portion of the data to produce the set of preliminary partial results, wherein the at least one other computing device producing at least one other preliminary partial result of the set of preliminary partial results;
generate first interim data based on at least some of the set of preliminary partial results;
execute the first unique partial sub-task on at least one of the first portion of the data or the first interim data to produce a first partial result;
generate first partial redundancy data based on the first interim data;
transmit, via the interface and to a second other computing device, the first partial redundancy data processed based on decoding by the second other computing device in accordance with a dispersed storage error coding function to produce redundancy data; and
transmit, via the interface and to at least one of the second other computing device or a third other computing device, the first partial result to be processed by the at least one of the second other computing device or the third other computing device to produce a result corresponding to at least one of the set of partial tasks performed on the data.
10. The computing device of claim 9 , wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:
generate the first interim data based on the first preliminary partial result of the set of preliminary partial results and one or more of the at least one other preliminary partial result of the set of preliminary partial results.
11. The computing device of claim 9 , wherein:
the data is allocated into the first portion of the data and the second portion of the data based on at least one of a time parameter, a geographic parameter, or a source parameter;
the common task corresponding to a function to identify a common characteristic of portions of the data including the first portion of the data; and
the first unique partial sub-task corresponding to a function to categorize uniquely the common characteristic of the portions of the data based on a unique parameter.
12. The computing device of claim 9 further comprising:
a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.
13. The computing device of claim 9 , wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).
14. A method for execution by a computing device, the method comprising:
receiving, via an interface of the computing device that is configured to interface and communicate with a dispersed or distributed storage network (DSN) and from a first other computing device, a first partial task of a set of partial tasks and a first portion of data, wherein the first partial task of the set of partial tasks includes a common task and a first unique partial sub-task;
executing the common task on the first portion of the data to produce a first preliminary partial result of a set of preliminary partial results;
generating first interim data based on at least some of the set of preliminary partial results;
executing the first unique partial sub-task on at least one of the first portion of the data or the first interim data to produce a first partial result;
generating first partial redundancy data based on the first interim data;
transmitting, via the interface and to a second other computing device, the first partial redundancy data processed based on decoding by the second other computing device in accordance with a dispersed storage error coding function to produce redundancy data; and
transmitting, via the interface, the first partial result to at least one of the second other computing device or a third other computing device to be processed by the at least one of the second other computing device or the third other computing device to produce a result corresponding to at least one of the set of partial tasks performed on the data.
15. The method of claim 14 further comprising:
executing the common task on the first portion of the data in cooperation with at least one other computing device executing the common task on at least one other portion of the data to produce the set of preliminary partial results, wherein the at least one other computing device producing at least one other preliminary partial result of the set of preliminary partial results.
16. The method of claim 15 further comprising:
generating the first interim data based on the first preliminary partial result of the set of preliminary partial results and one or more of the at least one other preliminary partial result of the set of preliminary partial results.
17. The method of claim 14 further comprising:
receiving, via the interface, the first partial task of the set of partial tasks and the first portion of the data in cooperation with at least one other computing device configured to receive a second partial task of the set of partial tasks and a second portion of the data, wherein the data is allocated into the first portion of the data and the second portion of the data based on at least one of a time parameter, a geographic parameter, or a source parameter.
18. The method of claim 14 , wherein:
the common task corresponding to a function to identify a common characteristic of portions of the data including the first portion of the data; and
the first unique partial sub-task corresponding to a function to categorize uniquely the common characteristic of the portions of the data based on a unique parameter.
19. The method of claim 14 , wherein the computing device includes a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.
20. The method of claim 14 , wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).