Efficient method to optimize distributed segment processing mechanism in dedupe systems by leveraging the locality principle
One example method includes receiving at a dedupe system, from a client, a request that comprises a set of fingerprints, where each fingerprint in the set corresponds to a particular data segment, filtering, at the dedupe system, the set of fingerprints into a set of unique fingerprints and a set of non-unique fingerprints, reading, at the dedupe system, from a container where copies of the non-unique fingerprints are stored, an additional set of non-unique fingerprints, sending, from the dedupe system to the client, a single response that comprises both the set of unique fingerprints and the additional set of non-unique fingerprints, and receiving from the client, at the dedupe system, data segments that respectively correspond to the unique fingerprints in the set of unique fingerprints, but no data segments corresponding to the non-unique fingerprints in the set of non-unique fingerprints are received by the dedupe system from the client.
1. A method, comprising:
sending, by a client to a dedupe system, a set of fingerprints for filtering by the dedupe system;
receiving, by the client from the dedupe system, fingerprints of the set of fingerprints that were identified by the dedupe system as unique fingerprints, and additional non-unique fingerprints from a set of containers of the dedupe system;
sending, by the client to the dedupe system, only unique segments for writing by the dedupe system, and the unique segments form a set;
receiving, by the client system, data from a backup dataset, and segmenting and fingerprinting the data from the backup dataset to create a next set of fingerprints;
filtering, by the client system, fingerprints in the next set of fingerprints, using additional existing fingerprints received earlier from the dedupe system, so as to identify unique segments of the next set; and
sending, by the client system to the dedupe system, a segment tree of the unique segments of the next set.
2. The method as recited in claim 1 , wherein any fingerprints identified, during the filtering, as matching fingerprints already present in a local in-memory list, are not sent by the client system to the dedupe system.
3. The method as recited in claim 2 , wherein the local in-memory list includes one or more non-unique fingerprints previously sent by the client system to the dedupe system.
4. The method as recited in claim 1 , wherein the segment tree is usable by the dedupe system to build file metadata for a file.
5. The method as recited in claim 1 , wherein an amount of the unique fingerprints sent by the dedupe system to the client is specified by the client.
6. The method as recited in claim 1 , wherein an amount of the unique fingerprints sent by the dedupe system to the client is a function of a value of an adaptive match rate that is determined by the client.
7. The method as recited in claim 1 , wherein the unique fingerprints, and the additional non-unique fingerprints from the set of containers of the dedupe system, are both received by the client in response to a single RPC (remote procedure call) issued by the client to the dedupe system.
8. The method as recited in claim 1 , the unique fingerprints, and the additional non-unique fingerprints received by the client from the dedupe system, are written by the client to an in-memory list locally maintained at the client.
9. The method as recited in claim 1 , wherein the segment tree is sent at an end of a data ingest process at the dedupe system.
10. The method as recited in claim 1 , wherein the unique segments are created by the client.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
sending, by a client to a dedupe system, a set of fingerprints for filtering by the dedupe system;
receiving, by the client from the dedupe system, fingerprints of the set of fingerprints that were identified by the dedupe system as unique fingerprints, and additional non-unique fingerprints from a set of containers of the dedupe system;
sending, by the client to the dedupe system, only unique segments for writing by the dedupe system, and the unique segments form a set;
receiving, by the client system, data from a backup dataset, and segmenting and fingerprinting the data from the backup dataset to create a next set of fingerprints;
filtering, by the client system, fingerprints in the next set of fingerprints, using additional existing fingerprints received earlier from the dedupe system, so as to identify unique segments of the next set; and
sending, by the client system to the dedupe system, a segment tree of the unique segments of the next set.
12. The non-transitory storage medium as recited in claim 11 , wherein any fingerprints identified, during the filtering, as matching fingerprints already present in a local in-memory list, are not sent by the client system to the dedupe system.
13. The non-transitory storage medium as recited in claim 2 , wherein the local in-memory list includes one or more non-unique fingerprints previously sent by the client system to the dedupe system.
14. The non-transitory storage medium as recited in claim 11 , wherein the segment tree is usable by the dedupe system to build file metadata for a file.
15. The non-transitory storage medium as recited in claim 11 , wherein an amount of the unique fingerprints sent by the dedupe system to the client is specified by the client.
16. The non-transitory storage medium as recited in claim 11 , wherein an amount of the unique fingerprints sent by the dedupe system to the client is a function of a value of an adaptive match rate that is determined by the client.
17. The non-transitory storage medium as recited in claim 11 , wherein the unique fingerprints, and the additional non-unique fingerprints from the set of containers of the dedupe system, are both received by the client in response to a single RPC (remote procedure call) issued by the client to the dedupe system.
18. The non-transitory storage medium as recited in claim 11 , the unique fingerprints, and the additional non-unique fingerprints received by the client from the dedupe system, are written by the client to an in-memory list locally maintained at the client.
19. The non-transitory storage medium as recited in claim 11 , wherein the segment tree is sent at an end of a data ingest process at the dedupe system.
20. The non-transitory storage medium as recited in claim 11 , wherein the unique segments are created by the client.