USING A NAMESPACE TO AUGMENT DE-DUPLICATION
Data to be de-duplicated for storage in a DSN memory is received. A source name is generated and associated with the data. The source name is generated based on contents of the data to be stored. Encoded data slices are generated from the data, and slice names are assigned based on the source name associated with the data being encoded. Distributed storage (DS) units are selected based on the slice names of the encoded data slices, and are assigned to de-duplicate and store the encoded data slices. The encoded data slices are transmitted to the selected DS units for de-duplication and storage of de-duplicated encoded data slices.
1 . A method for use in a distributed storage network (DSN), the method comprising:
receiving data to be de-duplicated for storage in a DSN memory;
generating a source name associated with the data, the source name based on contents of the data to be stored;
generating encoded data slices from the data, the encoded data slices having slice names assigned based on the source name associated with the data being encoded;
assigning selected distributed storage (DS) units to de-duplicate and store the encoded data slices, wherein DS units are selected based on the slice names of the encoded data slices; and
transmitting the encoded data slices to the selected DS units for de-duplication and storage of de-duplicated encoded data slices.
2 . The method of claim 1 , further comprising:
associating a data identifier of the data with the source name.
3 . The method of claim 1 , further comprising:
segmenting the data to produce a plurality of data segments;
encoding each of the plurality of data segments to generate a set of encoded data slices, wherein at least a read-threshold number of encoded data slices included in the set of encoded data slices is required to reconstruct the data;
generating a set of slice names to assign to each of the encoded data slices included in the set of encoded data slices; and
issuing a set of write slice requests to the selected DS units, the set of write slice requests including the set of encoded data slices, and the set of slice names.
4 . The method of claim 3 , further comprising
receiving, by a selected DS unit of the DSN memory, a write slice request of the set of write slice requests, the write slice request including a first encoded data slice and a slice name.
5 . The method of claim 4 , further comprising:
determining whether the first encoded data slice is a duplicate of a second encoded data slice already stored by the selected DS unit.
6 . The method of claim 5 , further comprising:
in response to determining that the first encoded data slice is not a duplicate of a second encoded data slice already stored by the selected DS unit, storing the first encoded data slice in the selected DS unit.
7 . The method of claim 1 , wherein:
each slice name includes a corresponding pillar index field entry, the source name, and a segment number.
8 . A distributed storage network (DSN) comprising:
a distributed storage (DS) processing module including a processor and associated memory;
a DSN memory coupled to the DS processing module, the DSN memory including a processor and associated memory, and further including a plurality of DS units;
the DS processing module configured to:
receive data to be de-duplicated for storage in the DSN memory;
generate a source name associated with the data, the source name based on contents of the data to be stored;
generate encoded data slices from the data, the encoded data slices having slice names assigned based on the source name associated with the data being encoded;
assign selected DS units to de-duplicate and store the encoded data slices, wherein DS units are selected based on the slice names of the encoded data slices; and
transmit the encoded data slices to the selected DS units for de-duplication and storage of de-duplicated encoded data slices.
9 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
associate a data identifier of the data with the source name.
10 . The distributed storage network (DSN) of claim 8 , wherein the DS processing module is further configured to:
segment the data to produce a plurality of data segments;
encode each of the plurality of data segments to generate a set of encoded data slices, wherein at least a read-threshold number of encoded data slices included in the set of encoded data slices is required to reconstruct the data;
generate a set of slice names to assign to each of the encoded data slices included in the set of encoded data slices; and
issue a set of write slice requests to the selected DS units, the set of write slice requests including the set of encoded data slices, and the set of slice names.
11 . The distributed storage network (DSN) of claim 10 , wherein the DSN memory is configured to:
receive, at a selected DS unit of the DSN memory, a write slice request of the set of write slice requests, the write slice request including a first encoded data slice and a slice name.
12 . The distributed storage network (DSN) of claim 11 , wherein the DSN memory is further configured to:
determine whether the first encoded data slice is a duplicate of a second encoded data slice already stored by the selected DS unit.
13 . The distributed storage network (DSN) of claim 12 , wherein the DSN memory is further configured to:
in response to determining that the first encoded data slice is not a duplicate of a second encoded data slice already stored by the selected DS unit, storing the first encoded data slice in the selected DS unit.
14 . The distributed storage network (DSN) of claim 8 , wherein:
each slice name includes a corresponding pillar index field entry, the source name, and a segment number.
15 . A distributed storage (DS) processing module comprising:
a processor;
memory coupled to the processor;
the processor configured to:
receive data to be de-duplicated for storage in a distributed storage network (DSN) memory, the DSN memory including a plurality of DS units configured to store encoded data slices;
generate a source name associated with the data, the source name based on contents of the data to be stored;
generate encoded data slices from the data, the encoded data slices having slice names assigned based on the source name associated with the data being encoded;
assign selected DS units to de-duplicate and store the encoded data slices, wherein DS units are selected based on the slice names of the encoded data slices; and
transmit the encoded data slices to the selected DS units for de-duplication and storage of de-duplicated encoded data slices.
16 . The distributed storage (DS) processing module of claim 15 , wherein the processor is further configured to:
associate a data identifier of the data with the source name.
17 . The distributed storage (DS) processing module of claim 16 , wherein the processor is further configured to:
associate the data identifier with the source name by updating a directory.
18 . The distributed storage (DS) processing module of claim 15 , wherein the processor is further configured to:
segment the data to produce a plurality of data segments;
encode each of the plurality of data segments to generate a set of encoded data slices, wherein at least a read-threshold number of encoded data slices included in the set of encoded data slices is required to reconstruct the data;
generate a set of slice names to assign to each of the encoded data slices included in the set of encoded data slices; and
issue a set of write slice requests to the selected DS units, the set of write slice requests including the set of encoded data slices, and the set of slice names.
19 . The distributed storage (DS) processing module of claim 18 , wherein:
a write slice request of the set of write slice requests includes a first encoded data slice and a slice name.
20 . The distributed storage (DS) processing module of claim 15 , wherein:
each slice name includes a corresponding pillar index field entry, the source name, and a segment number.