Internet based shared memory in a distributed computing system
A method includes sending, by a computing device of a distributed storage and task (DST) network, a plurality of sets of encoded data slices and index generation task information to a set of DST execution units. The method further includes receiving partial data indexes from the set of DST execution units. The method further includes generating a data index based on the partial data indexes and determining an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index. The method further includes partitioning the operational task into a set of partial tasks and sending the set of partial tasks to the set of DST execution units.
1. A method comprises:
sending, by a computing device of a distributed storage and task (DST) network, a plurality of sets of encoded data slices and index generation task information to a set of DST execution units of the DST network, wherein raw data is dispersed storage error encoded in accordance with initial dispersed storage error encoding parameters to produce the plurality of sets of encoded data slices, and wherein the index generation task information is regarding the raw data;
receiving, by the computing device, partial data indexes from the set of DST execution units, wherein the partial data indexes are generated from the plurality of sets of encoded data slices and based on the index generation task information;
generating, by the computing device, a data index based on the partial data indexes;
determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the plurality of sets of encoded data slices and execute a data processing function on the data index;
partitioning, by the computing device, the operational task into a set of partial tasks; and
sending, by the computing device, the set of partial tasks to the set of DST execution units.
2. The method of claim 1 further comprises:
determining, by the computing device, the index generation task information based on a storage profile regarding the raw data.
3. The method of claim 2 , wherein the index generation task information includes one or more of:
a search parameter;
a keyword;
pattern recognition information; and
timing information.
4. The method of claim 2 , wherein the storage profile includes one or more of:
a security requirement;
a performance requirement;
an estimated retrieval frequency;
an estimated distributed processing level;
a data visibility profile;
a data owner;
a data index storage indicator; and
a data deletion policy.
5. The method of claim 1 , wherein the data index includes one or more of:
identified keywords;
identified dates;
identified internet protocol addresses;
partial content;
a word count;
raw data statistics;
a summary of the raw data; and
one or more DST network addresses.
6. The method of claim 1 further comprises:
when the operational task is long term storage of the raw data, selecting, by the computing device, new dispersed storage error encoding parameters;
dispersed storage error encoding, by the computing device, the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and
sending, by the computing device, the new plurality of sets of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.
7. A computing device of a distributed storage and task (DST) network, the computing device comprises:
an interface;
memory; and
a processing module operably coupled to the memory and the interface, wherein the processing module is operable to:
send a plurality of sets of encoded data slices and index generation task information to a set of DST execution units of the DST network, wherein raw data is dispersed storage error encoded in accordance with initial dispersed storage error encoding parameters to produce the plurality of sets of encoded data slices, and wherein the index generation task information is regarding the raw data;
receiving, by the computing device, partial data indexes from the set of DST execution units, wherein the partial data indexes are generated from the plurality of sets of encoded data slices and based on the index generation task information;
generating, by the computing device, a data index based on the partial data indexes;
determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the plurality of sets of encoded data slices and execute a data processing function on the data index;
partitioning, by the computing device, the operational task into a set of partial tasks; and
sending, by the computing device, the set of partial tasks to the set of DST execution units.
8. The computing device of claim 7 , wherein the processing module is further operable to:
determine the index generation task information based on a storage profile regarding the raw data.
9. The computing device of claim 8 , wherein the index generation task information includes one or more of:
a search parameter;
a keyword;
pattern recognition information; and
timing information.
10. The computing device of claim 8 , wherein the storage profile includes one or more of:
a security requirement;
a performance requirement;
an estimated retrieval frequency;
an estimated distributed processing level;
a data visibility profile;
a data owner;
a data index storage indicator; and
a data deletion policy.
11. The computing device of claim 7 , wherein the data index includes one or more of:
identified keywords;
identified dates;
identified internet protocol addresses;
partial content;
a word count;
raw data statistics;
a summary of the raw data; and
one or more DST network addresses.
12. The computing device of claim 7 , wherein the processing module is further operable to:
when the operational task is long term storage of the raw data, select new dispersed storage error encoding parameters;
dispersed storage error encode the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and
send the new plurality of sets of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.