DE-DUPLICATION OF DATA STREAMS
A data segment is encrypted to produce an encrypted data segment, and a data tag associated with the data segment is generated. The encrypted data segment is encoded to generate a set of encoded data slices. At least a read-threshold number of encoded data slices are required to reconstruct the encrypted data segment. A set of write slice requests, which includes the set of encoded data slices and the data tag, is transmitted to a DSN memory. A determination is made, based on the data tag, whether a first encoded data slice of the set of encoded data slices is a duplicate of a second encoded data slice already stored within the DSN memory. If it is a duplicate, rather of storing the first encoded data slice, a reference to a location of the second encoded data slice is stored.
1 . A method for use in a distributed storage network (DSN), the method comprising:
encrypting a data segment to produce an encrypted data segment;
generating a data tag associated with the data segment;
encoding the encrypted data segment to generate a set of encoded data slices including a plurality of encoded data slices, wherein at least a read-threshold number of the plurality of encoded data slices included in the set of encoded data slices is required to reconstruct the encrypted data segment;
transmitting a set of write slice requests to a DSN memory, the set of write slice requests including the set of encoded data slices, and the data tag;
determining, based on the data tag, whether a first encoded data slice of the set of encoded data slices is a duplicate of a second encoded data slice already stored within the DSN memory; and
in response to determining that the first encoded data slice is a duplicate of the second encoded data slice, storing a reference to a location where the second encoded data slice is stored, instead of storing the first encoded data slice.
2 . The method of claim 1 , further comprising:
generating a key
masking the key to generate a masked key; and
including the masked key in the encrypted data segment prior to generating the set of encoded data slices.
3 . The method of claim 1 , further comprising:
generating a key
obfuscating the key to generate an obfuscated key; and
storing the obfuscated key as metadata associated with the data segment.
4 . The method of claim 1 , further comprising:
generating the data tag by applying a deterministic function to the data segment prior to encrypting the data segment.
5 . The method of claim 1 , further comprising:
generating the data tag by applying a deterministic function to the encrypted data segment.
6 . The method of claim 1 , further comprising:
in response to determining that the first encoded data slice is not duplicated within the DSN memory:
storing the first encoded data slice at a location associated with a distributed storage (DS) unit included in the DSN memory; and
outputting, to other DS units included in the DSN memory, information associating a storage location of the first encoded data slice with the data tag.
7 . The method of claim 1 , further comprising:
extracting the data tag from a write slice request to generate an extracted data tag; and
comparing the extracted data tag to a data tag list including a list of data tags associated with encoded data slices already stored in the DSN memory.
8 . A distributed storage network (DSN) comprising:
a distributed storage (DS) processing module including a processor and associated memory, the DS processing module configured to:
encrypt a data segment to produce an encrypted data segment;
generate a data tag associated with the data segment;
encode the encrypted data segment to generate a set of encoded data slices including a plurality of encoded data slices, wherein at least a read-threshold number of the plurality of encoded data slices included in the set of encoded data slices is required to reconstruct the encrypted data segment;
transmit a set of write slice requests to a DSN memory, the set of write slice requests including the set of encoded data slices, and the data tag;
a DSN memory including a processor and associated memory, and further including a plurality of DS units, the DSN memory configured to:
determine, based on the data tag, whether a first encoded data slice of the set of encoded data slices is a duplicate of a second encoded data slice already stored within the DSN memory; and
store error encoded data slices on behalf of the DSN, the DSN memory configured to, in response to determining that the first encoded data slice is a duplicate of the second encoded data slice, store a reference to a location where the second encoded data slice is stored, instead of storing the first encoded data slice.
9 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
generate a key
mask the key to generate a masked key; and
include the masked key in the encrypted data segment prior to generating the set of encoded data slices.
10 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
generate a key
obfuscate the key to generate an obfuscated key; and
store the obfuscated key as metadata associated with the data segment.
11 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
generate the data tag by applying a deterministic function to the data segment prior to encrypting the data segment.
12 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
generate the data tag by applying a deterministic function to the encrypted data segment.
13 . The distributed storage network (DSN) of claim 8 , wherein the DSN memory is further configured to:
in response to determining that the first encoded data slice is not duplicated within the DSN memory:
store the first encoded data slice at a location associated within a distributed storage (DS) unit; and
output, to other DS units included in the DSN memory, information associating a storage location of the first encoded data slice with the data tag.
14 . The distributed storage network (DSN) of claim 8 , wherein the DSN memory is further configured to:
extract the data tag from a write slice request to generate an extracted data tag; and
compare the extracted data tag to a data tag list including a list of data tags associated with encoded data slices already stored in the DSN memory.
15 . A distributed storage network (DSN) memory comprising:
a processor and associated memory;
a plurality of distributed storage (DS) units coupled to the processor and associated memory;
the processor and associated memory configured to:
receive a set of write slice requests from DS processing module, the set of write slice requests including a set of encoded data slices, and a data tag, wherein the set of encoded data slices includes a plurality of encoded data slices generated from an encrypted data segment, and wherein at least a read-threshold number of encoded data slices included in the set of encoded data slices is required to reconstruct the encrypted data segment;
determine, based on the data tag, whether a first encoded data slice of the set of encoded data slices is a duplicate of a second encoded data slice already stored by a DS unit included in the DSN memory; and
in response to determining that the first encoded data slice is a duplicate of the second encoded data slice, store a reference to a location where the second encoded data slice is stored, instead of storing the first encoded data slice.
16 . The distributed storage network (DSN) memory of claim 15 , wherein the set of write slice requests includes metadata, the metadata including an obfuscated key.
17 . The distributed storage network (DSN) memory of claim 15 , further configured to:
extract the data tag from a write slice request to generate an extracted data tag; and
compare the extracted data tag to a data tag list including a list of data tags associated with encoded data slices already stored in the DSN memory.
18 . The distributed storage network (DSN) memory of claim 15 , further configured to:
in response to determining that the first encoded data slice is not duplicated within the DSN memory:
store the first encoded data slice at a location associated within a distributed storage (DS) unit; and
output, to other DS units included in the DSN memory, information associating a storage location of the first encoded data slice with the data tag.
19 . The distributed storage network (DSN) memory of claim 18 , further configured to:
store metadata associated with the first encoded data slice in one of the plurality of distributed storage (DS) units included in the DSN memory.
20 . The distributed storage network (DSN) memory of claim 15 , wherein:
the reference to a location where the second encoded data slice is stored includes a reference to an alternate DS unit.