Data de-duplication in a dispersed storage network utilizing data characterization

1. A method for execution by a processing module of a computing device, the method comprises:

receiving, from a requesting device, a data storage request that includes data for storage;

determining, by the processing module, whether substantially identical data is currently stored in a dispersed storage network (DSN) memory as a plurality of sets of encoded data slices, wherein the identical data was encoded in accordance with a dispersed storage error encoding function to produce the plurality of sets of encoded data slices, and wherein the substantially identical data is recoverable based on a unique retrieval matrix of the plurality of sets of encoded data slices; and

when the substantially identical data is stored in the DSN memory:

generating, for the requesting device, a second unique retrieval matrix of the plurality of sets of encoded data slices, wherein the requesting device can recover at least a portion of the data based on the second unique retrieval matrix of the plurality of sets of encoded data slices, wherein:

the unique retrieval matrix including:

for a first set of encoded data slices of the plurality of sets of encoded data slices, identity of a first sub-set of encoded data slices of the first set of encoded data slices; and

for a second set of encoded data slices of the plurality of sets of encoded data slices, identity of a first sub-set of encoded data slices of the second set of encoded data slices; and

the second unique retrieval matrix including:

for the first set of encoded data slices of the plurality of sets of encoded data slices, identity of a second sub-set of encoded data slices of the first set of encoded data slices; and

for the second set of encoded data slices of the plurality of sets of encoded data slices, identity of a second sub-set of encoded data slices of the second set of encoded data slices, wherein each sub-set of encoded data slices includes a least a decode threshold number of encoded data slices.

2. The method of claim 1 further comprises:

the first sub-set of encoded data slices of the first set of encoded data slices including at least some encoded data slices having different pillar numbers than encoded data slices of the first sub-set of encoded data slices of the second set of encoded data slices; and

the second sub-set of encoded data slices of the first set of encoded data slices including at least some encoded data slices having different pillar numbers than encoded data slices of the second sub-set of encoded data slices of the second set of encoded data slices.

3. The method of claim 1 , wherein the determining whether the substantially identical data is currently stored in the DSN memory further comprises at least one of:

determining whether a data identifier associated with the data substantially matches a data identifier associated with the substantially identical data;

determining whether a calculated hash of the data substantially matches a stored hash of the substantially identical data; and

comparing the data to the substantially identical data.

4. The method of claim 1 further comprises:

generating, using a key generating function, an encryption key based on the data; and

comparing the encryption key with a stored encryption key of the substantially identical data, wherein the stored encryption key was generated, using the key generating function, based on the substantially identical data; and

when the encryption key compares favorably with the stored encryption key, indicating that the substantially identical data is currently stored.

5. The method of claim 4 , wherein the key generating function comprises at least one of:

setting the encryption key to be substantially equal to the data; and

generating a representation of the data and setting the encryption key to be substantially equal to the representation of the data, wherein the representation of the data is generated by a reversible transformation such equivalency, adding a constant, subtracting a constant, or inverting one or more bits.

6. The method of claim 4 further comprises:

generating a key reference based on the encryption key; and

comparing the encryption key with a stored encryption key by comparing the key reference with a stored key reference of the stored encryption key.

7. The method of claim 6 , wherein the generating the key reference comprises at least one of:

transforming the encryption key;

performing a hash function on the encryption key;

equating the key reference to a distributed stored network (DSN) address at which the encryption key is stored;

equating the key reference to information regarding the data, wherein the information includes one or more of: data object name, a data size, a hash of the data object, and indicators regarding the data.

8. A computing device comprises:

an interface; and

a processing module operable to:

receive, from a requesting device via the interface, a data storage request that includes data for storage;

determine whether substantially identical data is currently stored in a dispersed storage network (DSN) memory as a plurality of sets of encoded data slices, wherein the identical data was encoded in accordance with a dispersed storage error encoding function to produce the plurality of sets of encoded data slices, and wherein the substantially identical data is recoverable based on a unique retrieval matrix of the plurality of sets of encoded data slices; and

when the substantially identical data is stored in the DSN memory, generate, for the requesting device, a second unique retrieval matrix of the plurality of sets of encoded data slices, wherein the requesting device can recover at least a portion of the data based on the second unique retrieval matrix of the plurality of sets of encoded data slices, wherein:

the unique retrieval matrix including:

for a first set of encoded data slices of the plurality of sets of encoded data slices, identity of a first sub-set of encoded data slices of the first set of encoded data slices; and

for a second set of encoded data slices of the plurality of sets of encoded data slices, identity of a first sub-set of encoded data slices of the second set of encoded data slices; and

the second unique retrieval matrix including:

for the first set of encoded data slices of the plurality of sets of encoded data slices, identity of a second sub-set of encoded data slices of the first set of encoded data slices; and

for the second set of encoded data slices of the plurality of sets of encoded data slices, identity of a second sub-set of encoded data slices of the second set of encoded data slices, wherein each sub-set of encoded data slices includes a least a decode threshold number of encoded data slices.

9. The computing device of claim 8 further comprises:

the first sub-set of encoded data slices of the first set of encoded data slices including at least some encoded data slices having different pillar numbers than encoded data slices of the first sub-set of encoded data slices of the second set of encoded data slices; and

the second sub-set of encoded data slices of the first set of encoded data slices including at least some encoded data slices having different pillar numbers than encoded data slices of the second sub-set of encoded data slices of the second set of encoded data slices.

10. The computing device of claim 8 , wherein the processing module is further operable to determine whether the substantially identical data is currently stored in the DSN memory by at least one of:

determining whether a data identifier associated with the data substantially matches a data identifier associated with the substantially identical data;

determining whether a calculated hash of the data substantially matches a stored hash of the substantially identical data; and

comparing the data to the substantially identical data.

11. The computing device of claim 8 , wherein the processing module is further operable to:

generate, using a key generating function, an encryption key based on the data; and

compare the encryption key with a stored encryption key of the substantially identical data, wherein the stored encryption key was generated, using the key generating function, based on the substantially identical data; and

when the encryption key compares favorably with the stored encryption key, indicate that the substantially identical data is currently stored.

12. The computing device of claim 11 , wherein the key generating function comprises at least one of:

setting the encryption key to be substantially equal to the data; and

generating a representation of the data and setting the encryption key to be substantially equal to the representation of the data, wherein the representation of the data is generated by a reversible transformation such equivalency, adding a constant, subtracting a constant, or inverting one or more bits.

13. The computing device of claim 11 , wherein the processing module is further operable to:

generate a key reference based on the encryption key; and

compare the encryption key with the stored encryption key by comparing the key reference with a stored key reference of the stored encryption key.

14. The computing device of claim 13 , wherein the processing module is further operable to generate the key reference by at least one of:

transforming the encryption key;

performing a hash function on the encryption key;

equating the key reference to a distributed stored network (DSN) address at which the encryption key is stored;

equating the key reference to information regarding the data, wherein the information includes one or more of: data object name, a data size, a hash of the data object, and indicators regarding the data.