Combining deduplication with locality for efficient and fast storage
A method begins by receiving a storage request to store one or more portions of a data object and determining whether the one or more portions are stored within a dispersed storage network memory. When stored, the method continues by determining a retrieval demand for a portion of the one or more portions. When below a first threshold, the method continues by indicating that an existing copy of the one or more portions will be used for retrieval requests for the one or more portions. When above the first threshold, the method continues by copying the existing copy to create a first one or more portion copies and indicating the one or more portions will be used for retrieval requests from a first group of user computing devices and indicating the first one or more portion copies will be used for retrieval requests from a second group.
1. A method comprises:
receiving, by a computing device of a dispersed storage network (DSN), a storage request to store one or more portions of a data object, wherein the data object includes a plurality of portions;
determining, by the computing device, whether the one or more portions are already stored within a DSN memory;
when the one or more portions are stored:
determining, by the computing device, a retrieval demand for a portion of the one or more portions;
when the retrieval demand for the portion of the one or more portions is below a first threshold, indicating, by the computing device, that an existing copy of the one or more portions will be used for retrieval requests of the one or more portions within the DSN;
when the retrieval demand for the portion of the one or more portions is above the first threshold:
copying, by the computing device, the existing copy of the one or more portions to create a first one or more portion copies;
indicating, by the computing device, that the existing copy of the one or more portions will be used for retrieval requests from a first group of user computing devices of the DSN; and
indicating, by the computing device, that the first one or more portion copies will be used for retrieval requests from a second group of user computing devices of the DSN.
2. The method of claim 1 further comprises:
dividing the data object into a plurality of data segments, wherein one portion of the plurality of portions includes one or more data segments, and wherein the plurality of data segments is dispersed storage error encoded into pluralities of sets of encoded data slices.
3. The method of claim 1 , wherein the determining the retrieval demand comprises one or more of:
an anticipated retrieval demand level;
a historical retrieval demand level based on related content;
indicator from a content provider; and
indicator from a system operator.
4. The method of claim 1 further comprises:
when the retrieval demand for the portion of the one or more portions is above a second threshold, wherein the second threshold is greater than the first threshold:
copying, by the computing device, the existing copy for each of the one or more portions or the first one or more portion copies to create a second one or more portion copies; and
indicating, by the computing device, that the second one or more portion copies will be used for retrieval requests from a third group of user computing devices of the DSN.
5. The method of claim 1 further comprises:
receiving from a first user computing device, a retrieval request for a particular portion of the data object;
determining whether the particular portion has been replicated; and
when the particular portion has been replicated:
determining a set of storage units to retrieve the particular portion from based on the first user computing device affiliation with a group of user computing devices.
6. The method of claim 5 further comprises:
when the particular portion has not been replicated:
identifying a set of storage units within the DSN that is storing the particular portion.
7. The method of claim 5 further comprises:
retrieving one or more sets of encoded data slices from the set of storage units;
decoding the one or more sets of encoded data slices to produce the particular portion; and
sending the particular portion to the first user computing device.
8. A computing device of a dispersed storage network (DSN) comprises:
memory;
an interface; and
a processing module operably coupled to the memory and the interface, wherein the processing module is operable to:
receive, via the interface, a storage request to store one or more portions of a data object, wherein the data object includes a plurality of portions;
determine whether the one or more portions are already stored within a DSN memory;
when the one or more portions are stored:
determine a retrieval demand for a portion of the one or more portions;
when the retrieval demand for the portion of the one or more portions is below a first threshold, indicate that an existing copy of the one or more portions will be used for retrieval requests of the one or more portions within the DSN;
when the retrieval demand for the portion of the one or more portions is above the first threshold:
copy the existing copy of the one or more portions to create a first one or more portion copies;
indicate that the existing copy of the one or more portions will be used for retrieval requests from a first group of user computing devices of the DSN; and
indicate that the first one or more portion copies will be used for retrieval requests from a second group of user computing devices of the DSN.
9. The computing device of claim 8 , wherein the processing module is further operable to:
divide the data object into a plurality of data segments, wherein one portion of the plurality of portions includes one or more data segments, and wherein the plurality of data segments is dispersed storage error encoded into pluralities of sets of encoded data slices.
10. The computing device of claim 8 , wherein the processing module is operable to determine the retrieval demand by one or more of:
an anticipated retrieval demand level;
a historical retrieval demand level based on related content;
indicator from a content provider; and
indicator from a system operator.
11. The computing device of claim 8 , wherein the processing module is further operable to:
when the retrieval demand for the portion of the one or more portions is above a second threshold, wherein the second threshold is greater than the first threshold:
copy the existing copy for each of the one or more portions or the first one or more portion copies to create a second one or more portion copies; and
indicate that the second one or more portion copies will be used for retrieval requests from a third group of user computing devices of the DSN.
12. The computing device of claim 8 , wherein the processing module is further operable to:
receive, via the interface and from a first user computing device, a retrieval request for a particular portion of the data object;
determine whether the particular portion has been replicated; and
when the particular portion has been replicated:
determine a set of storage units to retrieve the particular portion from based on the first user computing device affiliation with a group of user computing devices.
13. The computing device of claim 12 , wherein the processing module is further operable to:
when the particular portion has not been replicated:
identify a set of storage units within the DSN that is storing the particular portion.
14. The computing device of claim 12 , wherein the processing module is further operable to:
retrieve, via the interface, one or more sets of encoded data slices from the set of storage units;
decoding the one or more sets of encoded data slices to produce the particular portion; and
sending, via the interface, the particular portion to the first user computing device.