Internet based shared memory in a distributed computing system

1. A method comprises:

receiving, by a computing device of a distributed storage and task (DST) network, raw data;

dispersed storage error encoding, by the computing device, the raw data in accordance with initial dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein the initial dispersed storage error encoding parameters include one or more of: initial data segmenting information, initial segment security information, initial error coding information, initial slicing information, and initial slice security information;

determining, by the computing device, index generation task information regarding the raw data;

sending, by the computing device, the plurality of sets of encoded data slices and the index generation task information to a set of DST execution units;

generating, by the set of DST execution units, partial data indexes from the plurality of sets of encoded data slices based on the index generation task information;

generating, by the computing device, a data index based on the partial data indexes;

determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index;

partitioning, by the computing device, the operational task into a set of partial tasks; and

sending, by the computing device, the set of partial tasks to the set of DST execution units.

2. The method of claim 1 further comprises:

executing, by at least some of the set of DST execution units, the set of partial tasks on at least some of the plurality of sets of encoded data slices.

3. The method of claim 1 , wherein the determining the index generation task information comprises:

determining, by the computing device, the index generation task information based on a storage profile regarding the raw data.

4. The method of claim 3 , wherein the index generation task information includes one or more of:

a search parameter;

a keyword;

pattern recognition information; and

timing information.

5. The method of claim 3 , wherein the storage profile includes one or more of:

a security requirement;

a performance requirement;

an estimated retrieval frequency;

an estimated distributed processing level;

a data visibility profile;

a data owner;

a data index storage indicator; and

a data deletion policy.

6. The method of claim 1 , wherein the data index includes one or more of:

identified keywords;

identified dates;

identified internet protocol addresses;

partial content;

a word count;

raw data statistics;

a summary of the raw data; and

one or more DST network addresses.

7. The method of claim 1 further comprises:

when the operational task is long term storage of the raw data, selecting, by the computing device, new dispersed storage error encoding parameters;

dispersed storage error encoding, by the computing device, the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and

sending, by the computing device, the new plurality of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.

8. A computer readable memory comprises:

a first memory element that stores operational instructions that, when executed by a computing device of a distributed storage and task (DST) network, causes the computing device to:

receive raw data;

dispersed storage error encode the raw data in accordance with initial dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein the initial dispersed storage error encoding parameters include one or more of: initial data segmenting information, initial segment security information, initial error coding information, initial slicing information, and initial slice security information;

determine index generation task information regarding the raw data; and

send the plurality of sets of encoded data slices and the index generation task information to a set of DST execution units;

a second memory element that stores operational instructions that, when executed by the set of DST execution units, causes the set of DST execution units to:

generate partial data indexes from the plurality of sets of encoded data slices based on the index generation task information;

a third memory element that stores operational instructions that, when executed by the computing device, causes the computing device to:

generate a data index based on the partial data indexes;

determine an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index;

partition the operational task into a set of partial tasks; and

send the set of partial tasks to the set of DST execution units.

9. The computer readable memory of claim 8 further comprises:

a fourth memory element that stores operational instructions that, when executed by the set of DST execution units, causes at least some of the set of DST execution units to:

execute the set of partial tasks on at least some of the plurality of sets of encoded data slices.

10. The computer readable memory of claim 8 , wherein the first memory element further stores operational instructions that, when executed by the computing device, causes the computing device to determine the index generation task information by:

determining the index generation task information based on a storage profile regarding the raw data.

11. The computer readable memory of claim 10 , wherein the index generation task information includes one or more of:

a search parameter;

a keyword;

pattern recognition information; and

timing information.

12. The computer readable memory of claim 10 , wherein the storage profile includes one or more of:

a security requirement;

a performance requirement;

an estimated retrieval frequency;

an estimated distributed processing level;

a data visibility profile;

a data owner;

a data index storage indicator; and

a data deletion policy.

13. The computer readable memory of claim 8 , wherein the data index includes one or more of:

identified keywords;

identified dates;

identified internet protocol addresses;

partial content;

a word count;

raw data statistics;

a summary of the raw data; and

one or more DST network addresses.

14. The computer readable memory of claim 8 , wherein the third memory element further stores operational instructions that, when executed by the computing device, causes the computing device to:

when the operational task is long term storage of the raw data select new dispersed storage error encoding parameters;

dispersed storage error encodes the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and

send the new plurality of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.