Efficient storage of encrypted data in a dispersed storage network
View Patent ↗A method begins with a processing module obtaining data to store and determining whether substantially similar data to the data is stored. When the substantially similar data is not stored, the method continues with the processing module generating a first encryption key based on the data, encoding the first encryption key into encoded data slices in accordance with an error coding dispersal storage function, and storing the encoded data slices in a dispersed storage network (DSN) memory. The method continues with the processing module encrypting the data using an encryption key of the substantially similar data in accordance with an encryption function to produce encrypted data, compressing the encrypted data in accordance with a compression function to produce compressed data, storing the compressed data when the substantially similar data is stored.
1. A method for execution by a processing module in a distributed storage (DS) unit, the method comprising:
receiving a request to store data from a requesting device;
identifying first substantially similar data to the data that is stored within dispersed storage network (DSN) memory, wherein the first substantially similar data is stored as first plurality of sets of error coded (EC) data slices;
comparing a number of other devices associated with the first substantially similar data to a threshold;
when the number is less than the threshold:
identifying, for the requesting device, first unique retrieval matrix of the first plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices based on the first unique retrieval matrix;
transmitting, via a communication interface of the DS unit, the first unique retrieval matrix to the requesting device; and
when the number is greater than or equal to the threshold:
generating second plurality of sets of EC data slices to store second substantially similar data to the data;
storing the second plurality of sets of EC data slices within the DSN memory;
determining, for the requesting device, second unique retrieval matrix of the second plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices based on the second unique retrieval matrix; and
transmitting, via the communication interface of the DS unit, the second unique retrieval matrix to the requesting device.
2. The method of claim 1 , wherein the identifying the first substantially similar data to the data that is stored within the DSN memory comprises one or more of:
identifying that a data identifier associated with the data substantially matches a data identifier associated with the first substantially similar data;
identifying that a calculated hash of the data substantially matches a stored hash of the first substantially similar data; and
identifying that the data compares favorably to the first substantially similar data.
3. The method of claim 1 , wherein the identifying the first unique retrieval matrix further comprises:
identifying one or more unique pillar combinations of one or more sets of the first plurality of sets of EC data slices that are unassigned; and
assigning at least one of the one or more unique pillar combinations of the one or more sets of the first plurality of sets of EC data slices to a user ID associated with the requesting device, wherein the first unique retrieval matrix corresponds to the at least one of the at least one of the one or more unique pillar combinations.
4. The method of claim 1 , wherein the generating the second plurality of sets of EC data slices further comprises:
generating the second plurality of sets of EC data slices based on one or more write operational parameters that include at least one write operational parameter that is not included in other one or more write operational parameters by which the second plurality of sets of EC data slices are generated.
5. The method of claim 1 , wherein the first unique retrieval matrix is based on an error coding dispersal storage function used to generate the first substantially similar data includes one or more of:
a pillars list;
a segmenting protocol;
a pre-slice data manipulation function;
a forward error correction encoding function
a slicing pillar width;
a post-slice data manipulation function;
a write threshold;
a read threshold; or
one or more unique sub-sets of the first plurality of sets of EC data slices.
6. The method of claim 1 , wherein the identifying the first unique retrieval matrix further comprises:
determining one or more read operational parameters for use in recovering the decode threshold number of EC data slices within the at least one set of the first plurality of sets of EC data slices; and
transmitting, via the communication interface of the DS unit, the one or more read operational parameters to the requesting device.
7. The method of claim 6 , wherein the one or more read operational parameters is based on one or more of:
an estimation of a number of common requesting devices;
an actual number of common requesting devices;
a subscription level indicator of the requesting device;
a number of sets within the first plurality of sets of EC data slices;
a previous read operational parameter;
a user ID;
the request to store data;
a data object name;
a data size;
a data type;
a data object;
a data object hash;
a priority indicator;
a security indicator; or
a performance indicator.
8. The method of claim 1 , wherein:
the first unique retrieval matrix identifies first unique sub-set of the first plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices and first pillar width of an EC dispersal storage function of the first plurality of sets of EC data slices; and
the second unique retrieval matrix identifies second unique sub-set of the second plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices and second pillar width of an EC dispersal storage function of the second plurality of sets of EC data slices.
9. A distributed storage (DS) processing unit comprises:
a network interface; and
a processing module configured to:
receive a request, via the network interface, to store data from a requesting device;
identify first substantially similar data to the data that is stored within dispersed storage network (DSN) memory, wherein the first substantially similar data is stored as first plurality of sets of error coded (EC) data slices;
compare a number of other devices associated with the first substantially similar data to a threshold;
when the number is less than the threshold:
identify, for the requesting device, first unique retrieval matrix of the first plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices based on the first unique retrieval matrix;
transmit, via a communication interface, the first unique retrieval matrix to the requesting device; and
when the number is greater than or equal to the threshold:
generate second plurality of sets of EC data slices to store second substantially similar data to the data;
store the second plurality of sets of EC data slices within the DSN memory;
determine, for the requesting device, second unique retrieval matrix of the second plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices based on the second unique retrieval matrix; and
transmit, via the communication interface of the DS unit, the second unique retrieval matrix to the requesting device.
10. The DS processing unit of claim 9 , wherein the processing module is further configured to:
identify the first substantially similar data by identifying that a data identifier associated with the data substantially matches a data identifier associated with the first substantially similar data;
identify the first substantially similar data by identifying that a calculated hash of the data substantially matches a stored hash of the first substantially similar data; or
identify the first substantially similar data by identifying that the data compares favorably to the first substantially similar data.
11. The DS processing unit of claim 9 , wherein the processing module is further configured to:
identify one or more unique pillar combinations of one or more sets of the first plurality of sets of EC data slices that are unassigned; and
assign at least one of the one or more unique pillar combinations of the one or more sets of the first plurality of sets of EC data slices to a user ID associated with the requesting device, wherein the first unique retrieval matrix corresponds to the at least one of the at least one of the one or more unique pillar combinations.
12. The DS processing unit of claim 9 , wherein the processing module is further configured to:
generate the second plurality of sets of EC data slices based on one or more write operational parameters that include at least one write operational parameter that is not included in other one or more write operational parameters by which the second plurality of sets of EC data slices are generated.
13. The DS processing unit of claim 9 , wherein the first unique retrieval matrix is based on an error coding dispersal storage function used to generate the first substantially similar data includes one or more of:
a pillars list;
a segmenting protocol;
a pre-slice data manipulation function;
a forward error correction encoding function
a slicing pillar width;
a post-slice data manipulation function;
a write threshold;
a read threshold; or
one or more unique sub-sets of the first plurality of sets of EC data slices.
14. The DS processing unit of claim 9 , wherein the processing module is further configured to:
determine one or more read operational parameters for use in recovering the decode threshold number of EC data slices within the at least one set of the first plurality of sets of EC data slices; and
transmit, via the communication interface of the DS unit, the one or more read operational parameters to the requesting device.
15. The DS processing unit of claim 9 , wherein:
the first unique retrieval matrix identifies first unique sub-set of the first plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices and first pillar width of an EC dispersal storage function of the first plurality of sets of EC data slices; and
the second unique retrieval matrix identifies second unique sub-set of the second plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices and second pillar width of an EC dispersal storage function of the second plurality of sets of EC data slices.
16. A non-transitory computer readable medium having instructions causing a processing module in a distributed storage (DS) unit to execute a method comprising:
receiving a request to store data from a requesting device;
identifying first substantially similar data to the data that is stored within dispersed storage network (DSN) memory, wherein the first substantially similar data is stored as first plurality of sets of error coded (EC) data slices;
comparing a number of other devices associated with the first substantially similar data to a threshold;
when the number is less than the threshold:
identifying, for the requesting device, first unique retrieval matrix of the first plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices based on the first unique retrieval matrix;
transmitting, via a communication interface of the DS unit, the first unique retrieval matrix to the requesting device; and
when the number is greater than or equal to the threshold:
generating second plurality of sets of EC data slices to store second substantially similar data to the data;
storing the second plurality of sets of EC data slices within the DSN memory;
determining, for the requesting device, second unique retrieval matrix of the second plurality of sets of EC data slices, wherein the requesting device can recover a decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices based on the second unique retrieval matrix; and
transmitting, via the communication interface of the DS unit, the second unique retrieval matrix to the requesting device.
17. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16 , wherein the identifying the first substantially similar data to the data that is stored within the DSN memory comprises one or more of:
identifying that a data identifier associated with the data substantially matches a data identifier associated with the first substantially similar data;
identifying that a calculated hash of the data substantially matches a stored hash of the first substantially similar data; and
identifying that the data compares favorably to the first substantially similar data.
18. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16 , wherein the identifying the first unique retrieval matrix further comprises:
identifying one or more unique pillar combinations of one or more sets of the first plurality of sets of EC data slices that are unassigned; and
assigning at least one of the one or more unique pillar combinations of the one or more sets of the first plurality of sets of EC data slices to a user ID associated with the requesting device, wherein the first unique retrieval matrix corresponds to the at least one of the at least one of the one or more unique pillar combinations.
19. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16 , wherein the generating the second plurality of sets of EC data slices further comprises:
generating the second plurality of sets of EC data slices based on one or more write operational parameters that include at least one write operational parameter that is not included in other one or more write operational parameters by which the second plurality of sets of EC data slices are generated.
20. The non-transitory computer readable medium having instructions causing the processing module to execute the method of claim 16 , wherein:
the first unique retrieval matrix identifies first unique sub-set of the first plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the first plurality of sets of EC data slices and first pillar width of an EC dispersal storage function of the first plurality of sets of EC data slices; and
the second unique retrieval matrix identifies second unique sub-set of the second plurality of sets of EC data slices that includes between the decode threshold number of EC data slices within at least one set of the second plurality of sets of EC data slices and second pillar width of an EC dispersal storage function of the second plurality of sets of EC data slices.