Efficient storage of data in a dispersed storage network
A data segment is encrypted to produce an encrypted data segment, and a data tag associated with the data segment is generated. The encrypted data segment is encoded to generate a set of encoded data slices. At least a read-threshold number of encoded data slices are required to reconstruct the encrypted data segment. A set of write slice requests, which includes the set of encoded data slices and the data tag, is transmitted to a DSN memory. A determination is made, based on the data tag, whether a first encoded data slice of the set of encoded data slices is a duplicate of a second encoded data slice already stored within the DSN memory. If it is a duplicate, rather of storing the first encoded data slice, a reference to a location of the second encoded data slice is stored.
1. A method comprising:
receiving, at a computing device including a processor, first data to be stored in a memory accessible to the processor;
identifying a data type associated with the first data;
generating a fingerprint of at least a portion of the first data;
determining, based on both the data type and the fingerprint, whether the first data is a duplicate of previously stored data; and
storing an object-linking file in response to determining that the first data is a duplicate of the previously stored data, wherein at least a portion of the object-linking file is stored in a same memory location where at least a portion of the first data would have been stored if the first data had not been a duplicate of the previously stored data, and wherein storing the object-linking file includes;
encoding the object-linking file using an error coding function to produce a set of encoded data slices; and
storing the set of encoded data slices in the memory.
2. The method of claim 1 , further comprising:
storing the first data in the memory in response to determining that the first data is not a duplicate of the previously stored data.
3. The method of claim 1 , further comprising:
generating a first data tag based on both the data type and the fingerprint; and
associating the first data tag with the first data.
4. The method of claim 3 , wherein associating the first data tag with the first data includes:
updating information included in one or more of a data tag list, a dispersed hierarchical index, or a distributed storage directory.
5. The method of claim 4 , wherein the information includes:
one or more of a data identifier of the first data, an address of the first data, or the first data tag.
6. A storage module comprising:
one or more storage units;
a communications interface configured to receive first data to be stored in the one or more storage units;
a computing core configured to:
identify a data type associated with the first data;
generate a fingerprint of at least a portion of the first data;
determine, based on both the data type and the fingerprint, whether the first data is a duplicate of previously stored data in the one or more storage units; and
store an object-linking file in response to a determination indicating that the first data is a duplicate of previously stored data in the one or more storage units, wherein at least a portion of the object-linking file is stored in a same memory location where at least a portion of the first data would have been stored if the first data had not been a duplicate of the previously stored data, wherein storing the object-linking file includes:
encoding the object-linking file using an error coding function to produce a set of encoded data slices; and
storing the set of encoded data slices in the one or more storage units.
7. The storage module of claim 6 , wherein the computing core is further configured to:
store the first data in the one or more storage units in response to determining that the first data is not a duplicate of the previously stored data.
8. The storage module of claim 6 , wherein the computing core is further configured to:
generate a first data tag based on both the data type and the fingerprint; and
associate the first data tag with the first data.
9. The storage module of claim 8 , wherein associating the first data tag with the first data includes:
updating information included in one or more of a data tag list, a dispersed hierarchical index, or a distributed storage directory.
10. The storage module of claim 9 , wherein the information includes:
one or more of a data identifier of the first data, an address of the first data, or the first data tag.
11. A non-transitory computer readable medium tangibly embodying a program of instructions to be stored in a memory and executed by a processor, the computer readable medium comprising:
at least one instruction to receive, at a computing device including a processor, first data to be stored in a memory accessible to the processor;
at least one instruction to identify a data type associated with the first data;
at least one instruction to generate a fingerprint of at least a portion of the first data;
at least one instruction to determine, based on both the data type and the fingerprint, whether the first data is a duplicate of previously stored data; and
at least one instruction to store an object-linking file in response to a result of the determining indicating that the first data is a duplicate of the previously stored data, wherein at least a portion of the object-linking file is stored in a same memory location where a corresponding portion of the first data would have been stored if the first data had not been a duplicate of the previously stored data, and wherein storing the object-linking file includes encoding the object-linking file using an error coding function to produce a set of encoded data slices and storing the set of encoded data slices in the memory.
12. The non-transitory computer readable medium of claim 11 , wherein at least one instruction to conditionally store includes:
at least one instruction to store the first data in the memory in response to determining that the first data is not a duplicate of the previously stored data.
13. The non-transitory computer readable medium of claim 11 , further comprising:
at least one instruction to generate a first data tag based on both the data type and the fingerprint; and
at least one instruction to associate the first data tag with the first data.
14. The non-transitory computer readable medium of claim 13 , wherein the at least one instruction to associate the first data tag with the first data includes:
at least one instruction to update information included in one or more of a data tag list, a dispersed hierarchical index, or a distributed storage directory.