Methods and apparatus for network efficient deduplication
View Patent ↗Mechanisms are provided for performing network efficient deduplication. Segments are extracted from files received for deduplication at a host connected to a target over one or more networks and/or fabrics in a deduplication system. Segment identifiers (IDs) are determined and compared with segment IDs for segments already deduplicated. Segments already deduplicated need not be transmitted to a target system. References and reference counts are modified at a target system. Updating references and reference counts may involve modifying filemaps, dictionaries, and datastore suitcases for both already deduplicated and not already deduplicated segments.
1. A method, comprising:
receiving a request at a host to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network and communicate using file level protocol;
determining segment boundaries for the plurality of files using data type information associated with the plurality of files and determine segment size using file type information;
determining whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system, wherein determining whether segments in the plurality of files have already been deduplicated includes comparing hashes of data segments stored at the target system with hashes in a deduplication dictionary at a source device, wherein ranges for the hashes of the data segments are assigned to different nodes within a cluster, each node corresponding to a separate deduplication dictionary, wherein if a first node is processing a data segment which has a hash value which maps to a second node, the first node will contact the second node that owns the range to find out if the data segments already exist in a datastore on the second node, wherein a work portioning scheme is applied based the hash ranges;
transmitting a first plurality of segments to the target system, the first plurality of segments not already deduplicated at the target system, the first plurality of segments added to a datastore suitcase, and not transmitting a second plurality of segments to the target system, the second plurality of segments already deduplicated at the target system;
modifying references and reference counts including filemaps and the datastore suitcases at the target system for the first plurality of segments and a second plurality of segments.
2. The method of claim 1 , wherein the plurality of segments span a plurality of files.
3. The method of claim 1 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
4. The method of claim 1 , wherein the second plurality of segments already deduplicated correspond to a plurality of filemaps.
5. The method of claim 1 , wherein the second plurality of segments already deduplicated correspond to a plurality of datastore suitcases.
6. The method of claim 5 , wherein a datastore suitcase in the plurality of datastore suitcases further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
7. The method of claim 1 , wherein determining whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system comprises comparing segment identifiers (segment IDs) for the segments match segment IDs for segments already deduplicated.
8. The method of claim 7 , wherein segment IDs are evaluated using a segment ID database at the host.
9. The method of claim 7 , wherein segment IDs are evaluated by transmitting segment IDs to the target system and comparing the segment IDs with entries in a deduplication dictionary.
10. An apparatus, comprising:
an input interface configured to receive a request to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network and communicate using file level protocol;
a processor configured to determine segment boundaries for the plurality of files using data type information associated with the plurality of files and determine segment size using file type information and determine whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system, wherein determining whether segments in the plurality of files have already been deduplicated includes comparing hashes of data segments stored at the target system with hashes in a deduplication dictionary at a source device, wherein ranges for the hashes of the data segments are assigned to different nodes within a cluster, each node corresponding to a separate deduplication dictionary, wherein if a first node is processing a data segment which has a hash value which maps to a second node, the first node will contact the second node that owns the range to find out if the data segments already exist in a datastore on the second node, wherein a work portioning scheme is applied based the hash ranges;
an output interface configured to transmit a first plurality of segments to the target system, the first plurality of segments not already deduplicated at the target system, the first plurality of segments added to a datastore suitcase, and not transmitting a second plurality of segments to the target system, the second plurality of segment already deduplicated at the target system;
wherein references and reference counts including filemaps and the datastore suitcases at the target system are modified for the first plurality of segments and a second plurality of segments.
11. The apparatus of claim 10 , wherein the plurality of segments span a plurality of files.
12. The apparatus of claim 10 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
13. The apparatus of claim 10 , wherein the second plurality of segments already deduplicated correspond to a plurality of filemaps.
14. The apparatus of claim 10 , wherein the second plurality of segments already deduplicated correspond to a plurality of datastore suitcases.
15. The apparatus of claim 14 , wherein a datastore suitcase in the plurality of datastore suitcases further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
16. The apparatus of claim 10 , wherein determining whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system comprises comparing segment identifiers (segment IDs) for the segments match segment IDs for segments already deduplicated.
17. The apparatus of claim 16 , wherein segment IDs are evaluated using a segment ID database at the host.
18. The apparatus of claim 16 , wherein segment IDs are evaluated by transmitting segment IDs to the target system and comparing the segment IDs with entries in a deduplication dictionary.
19. A non-transitory computer readable medium, comprising:
computer code for receiving a request at a host to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network and communicate using file level protocol;
computer code for determining segment boundaries for the plurality of files using data type information associated with the plurality of files and determine segment size using file type information;
computer code for determining whether segments in the plurality of files have already been deduplicated at a target system in a deduplication system, wherein determining whether segments in the plurality of files have already been deduplicated includes comparing hashes of data segments stored at the target system with hashes in a deduplication dictionary at a source device, wherein ranges for the hashes of the data segments are assigned to different nodes within a cluster, each node corresponding to a separate deduplication dictionary, wherein if a first node is processing a data segment which has a hash value which maps to a second node, the first node will contact the second node that owns the range to find out if the data segments already exist in a datastore on the second node, wherein a work portioning scheme is applied based the hash ranges;
computer code for transmitting a first plurality of segments to the target system, the first plurality of segments not already deduplicated at the target system, the first plurality of segments added to a datastore suitcase, and not transmitting a second plurality of segments to the target system, the second plurality of segment already deduplicated at the target system;
computer code for modifying references and reference counts including filemaps and the datastore suitcases at the target system for the first plurality of segments and a second plurality of segments.