Efficiently storing data in a dispersed storage network

1. A method for execution by a dispersed storage (DS) unit of a dispersed storage network (DSN), the method comprises:

ingesting a respective section of data, wherein a large amount of data is divided into sets of partitions, wherein each of the sets of partitions is divided into a set of sections, wherein a set of DS units ingests the set of sections, and wherein the set of DS units includes the DS unit;

for the ingested respective section of data:

dividing the ingested respective section of data into a plurality of data segments;

for a data segment of the plurality of data segments:

dividing the data segment into a plurality of data blocks;

temporarily storing the plurality of data blocks;

determining whether to not permanently store one or more of the plurality of data blocks;

when the one or more of the plurality of data blocks is to be permanently stored:

storing the one or more of the plurality of data blocks; and

generating a group of partial redundancy data based on the one or more of the plurality of data blocks and in accordance with a dispersed storage error coding function; and

when the one or more of the plurality of data blocks is not to be permanently stored:

creating a new plurality of data blocks from data blocks of the plurality of data blocks that are to be permanently stored and data blocks from another data segment that are to be permanently stored; and

generating the group of partial redundancy data based on the new plurality of data blocks in accordance with the dispersed storage error coding function.

2. The method of claim 1 , wherein the determining whether to not permanently store the one or more of the plurality of data blocks comprises:

determining that at least one other DS unit of the set of DS units has ingested a respective section that is substantially similar to the ingested respective section; and

in response to the determining that the at least one other DS unit has ingested the respective section that is substantially similar to the ingested respective section:

determining whether a de-duplication function is to be applied to the ingested respective section;

when the de-duplication function is to be applied, identifying the one or more of the plurality of data blocks that are not to be permanently stored; and

when the de-duplication function is not to be applied, indicating that the one or more of the plurality of data blocks is to be permanently stored.

3. The method of claim 1 , wherein the determining whether to not permanently store the one or more of the plurality of data blocks comprises:

analyzing the ingested respective section in accordance with data analysis criteria; and

when the analysis of the ingested respective section is unfavorable, identifying the one or more of the plurality of data blocks that are not to be permanently stored.

4. The method of claim 1 , wherein the generating the group of partial redundancy data based on the one or more of the plurality of data blocks comprises:

generating a first partial redundancy data for a first redundancy DS unit based on at least some of the plurality of data blocks and a first encoding parameter of the dispersed storage error coding function; and

generating a second partial redundancy data for a second redundancy DS unit based on at least another some of the plurality of data blocks and a second encoding parameter of the dispersed storage error coding function.

5. The method of claim 1 , wherein the creating the new plurality of data blocks comprises:

identifying the other data segment as a data segment for which partial redundancy data does not yet exist.

6. The method of claim 1 further comprises:

permanently storing the new plurality of data blocks.

7. A dispersed storage (DS) module of a dispersed storage (DS) unit of a dispersed storage network (DSN), the DS module comprises:

a first module, when operable within a computing device, causes the computing device to:

ingest a respective section of data, wherein a large amount of data is divided into sets of partitions, wherein each of the sets of partitions is divided into a set of sections, wherein a set of DS units ingests the set of sections, and wherein the set of DS units includes the DS unit;

a second module, when operable within the computing device, causes the computing device to:

for the ingested respective section of data:

divide the ingested respective section of data into a plurality of data segments;

for a data segment of the plurality of data segments:

divide the data segment into a plurality of data blocks;

temporarily store the plurality of data blocks; and

determine whether to not permanently store one or more of the plurality of data blocks;

a third module, when operable within the computing device, causes the computing device to:

when the one or more of the plurality of data blocks is to be permanently stored:

store the one or more of the plurality of data blocks; and

generate a group of partial redundancy data based on the one or more of the plurality of data blocks and in accordance with a dispersed storage error coding function; and

a fourth module, when operable within the computing device, causes the computing device to:

when the one or more of the plurality of data blocks is not to be permanently stored:

create a new plurality of data blocks from data blocks of the plurality of data blocks that are to be permanently stored and data blocks from another data segment that are to be permanently stored; and

generate the group of partial redundancy data based on the new plurality of data blocks in accordance with the dispersed storage error coding function.

8. The DS module of claim 7 , wherein the second module functions to determine whether to not permanently store the one or more of the plurality of data blocks by:

determining that at least one other DS unit of the set of DS units has ingested a respective section that is substantially similar to the ingested respective section; and

in response to the determining that the at least one other DS unit has ingested the respective section that is substantially similar to the ingested respective section:

determining whether a de-duplication function is to be applied to the ingested respective section;

when the de-duplication function is to be applied, identifying the one or more of the plurality of data blocks that are not to be permanently stored; and

when the de-duplication function is not to be applied, indicating that the one or more of the plurality of data blocks is to be permanently stored.

9. The DS module of claim 7 , wherein the second module functions to determine whether to not permanently store the one or more of the plurality of data blocks by:

analyzing the ingested respective section in accordance with data analysis criteria; and

when the analysis of the ingested respective section is unfavorable, identifying the one or more of the plurality of data blocks that are not to be permanently stored.

10. The DS module of claim 7 , wherein the third module functions to generate the group of partial redundancy data based on the one or more of the plurality of data blocks by:

generating a first partial redundancy data for a first redundancy DS unit based on at least some of the plurality of data blocks and a first encoding parameter of the dispersed storage error coding function; and

generating a second partial redundancy data for a second redundancy DS unit based on at least another some of the plurality of data blocks and a second encoding parameter of the dispersed storage error coding function.

11. The DS module of claim 7 , wherein the fourth module functions to create the new plurality of data blocks by:

identifying the other data segment as a data segment for which partial redundancy data does not yet exist.

12. The DS module of claim 7 further comprises:

the fourth module further functions to permanently store the new plurality of data blocks.