Efficient method to optimize distributed segment processing mechanism in dedupe systems by leveraging the locality principle
One example method includes receiving at a dedupe system, from a client, a request that comprises a set of fingerprints, where each fingerprint in the set corresponds to a particular data segment, filtering, at the dedupe system, the set of fingerprints into a set of unique fingerprints and a set of non-unique fingerprints, reading, at the dedupe system, from a container where copies of the non-unique fingerprints are stored, an additional set of non-unique fingerprints, sending, from the dedupe system to the client, a single response that comprises both the set of unique fingerprints and the additional set of non-unique fingerprints, and receiving from the client, at the dedupe system, data segments that respectively correspond to the unique fingerprints in the set of unique fingerprints, but no data segments corresponding to the non-unique fingerprints in the set of non-unique fingerprints are received by the dedupe system from the client.
1. A method, comprising:
receiving at a dedupe system, from a client, a request that comprises a set of fingerprints, where each fingerprint in the set corresponds to a particular data segment;
filtering, at the dedupe system, the set of fingerprints into a set of unique fingerprints and a set of non-unique fingerprints;
determining a container in a set of containers in the dedupe system where copies of the non-unique fingerprints are stored;
reading, at the dedupe system, from the container, an additional set of non-unique fingerprints, which exist in the container and are additional to the set of non-unique fingerprints, wherein the fingerprints in the additional set of non-unique fingerprints are selected based on a likelihood that those fingerprints would be requested in a future request by the client;
sending, from the dedupe system to the client in response to the request comprising the set of fingerprints, a single response that comprises both the set of unique fingerprints and the additional set of non-unique fingerprints, wherein a number of fingerprints in the single response is predetermined; and
receiving from the client, at the dedupe system, data segments that respectively correspond to the unique fingerprints in the set of unique fingerprints, but no data segments corresponding to the non-unique fingerprints in the set of non-unique fingerprints are received by the dedupe system from the client.
2. The method as recited in claim 1 , wherein all of the fingerprints in the response are sent in response to a single request from the client.
3. The method as recited in claim 1 , wherein a number of fingerprints in the additional set of non-unique fingerprints is specified in the request.
4. The method as recited in claim 1 , wherein some of the non-unique fingerprints in the additional set of non-unique fingerprints match fingerprints stored at the client.
5. The method as recited in claim 1 , further comprising writing, by the dedupe system, the data segments that respectively correspond to the unique fingerprints, and adding, by the dedupe system, the unique fingerprints to an index or database of the dedupe system.
6. The method as recited in claim 1 , wherein filtering comprises comparing the fingerprints contained in the request against a fingerprint filter.
7. The method as recited in claim 1 , wherein the fingerprints in the additional set of non-unique fingerprints do not necessitate an additional request from the client.
8. The method as recited in claim 1 , further comprising receiving, by the dedupe system from the client, a segment tree or metadata mapping associated with the data segments that respectively correspond to the unique fingerprints.
9. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising: receiving at a dedupe system, from a client, a request that comprises a set of fingerprints, where each fingerprint in the set corresponds to a particular data segment;
filtering, at the dedupe system, the set of fingerprints into a set of unique fingerprints and a set of non-unique fingerprints;
determining a container in a set of containers in the dedupe system where copies of the non-unique fingerprints are stored;
reading, at the dedupe system, from the container, an additional set of non-unique fingerprints, which exist in the container and are additional to the set of non-unique fingerprints, wherein the fingerprints in the additional set of non-unique fingerprints are selected based on a likelihood that those fingerprints would be requested in a future request by the client;
sending, from the dedupe system to the client in response to the request comprising the set of fingerprints, a single response that comprises both the set of unique fingerprints and the additional set of non-unique fingerprints, wherein a number of fingerprints in the single response is predetermined; and
receiving from the client, at the dedupe system, data segments that respectively correspond to the unique fingerprints in the set of unique fingerprints, but no data segments corresponding to the non-unique fingerprints in the set of non-unique fingerprints are received by the dedupe system from the client.
10. The non-transitory storage medium as recited in claim 9 , wherein all of the fingerprints in the response are sent in response to a single request from the client.
11. The non-transitory storage medium as recited in claim 9 , wherein a number of fingerprints in the additional set of non-unique fingerprints is specified in the request.
12. The non-transitory storage medium as recited in claim 9 , wherein some of the non-unique fingerprints in the additional set of non-unique fingerprints match fingerprints stored at the client.
13. The non-transitory storage medium as recited in claim 9 , wherein the operations further comprise writing, by the dedupe system, the data segments that respectively correspond to the unique fingerprints, and adding, by the dedupe system, the unique fingerprints to an index or database of the dedupe system.
14. The non-transitory storage medium as recited in claim 9 , wherein filtering comprises comparing the fingerprints contained in the request against a fingerprint filter.
15. The non-transitory storage medium as recited in claim 9 , wherein the fingerprints in the additional set of non-unique fingerprints do not necessitate an additional request from the client.
16. The non-transitory storage medium as recited in claim 9 , wherein the operations further comprise receiving, by the dedupe system from the client, a segment tree or metadata mapping associated with the data segments that respectively correspond to the unique fingerprints.