Methods and apparatus for network efficient deduplication
Mechanisms are provided for performing network efficient deduplication. Segments are extracted from files received for deduplication at a host connected to a target over one or more networks and/or fabrics in a deduplication system. Segment identifiers (IDs) are determined and compared with segment IDs for segments already deduplicated. Segments already deduplicated need not be transmitted to a target system. References and reference counts are modified at a target system. Updating references and reference counts may involve modifying filemaps, dictionaries, and datastore suitcases for both already deduplicated and not already deduplicated segments.
1. A method, comprising:
receiving a request at a host to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network, wherein the host communicates over the network with the target system by using a file level protocol, wherein the target system is a file server;
determining at the host whether segments in the plurality of files have already been deduplicated at the target system in a deduplication system by comparing segment identifiers (segment IDs) for the segments to match segment IDs for segments already deduplicated, wherein segment IDs are evaluated by transmitting segment IDs to the target system to compare the segment IDs with entries in a deduplication dictionary;
transmitting a first plurality of segments over a network to the target system by using the file level protocol, the first plurality of segments not already deduplicated at the target system, and not transmitting a second plurality of segments to the target system to reduce network bandwidth utilization, the second plurality of segments already deduplicated at the target system;
modifying references and reference counts at the target system for the first plurality of segments and the second plurality of segments, wherein modifying references and reference counts includes modifying a filemap, dictionaries, and a datastore suitcase for both already deduplicated and not already deduplicated segments;
wherein the dictionary is used to identify duplicate data segments and point to the location of the data segments, the datastore suitcase includes an index portion and a data portion to allow a system to perform a bulk read of the index portion to obtain offset data to allow parallel reads of large amounts of data in the data section and further includes three offset, reference count pairs which map to the data segments of the filemap file.
2. The method of claim 1 , wherein the plurality of segments span a plurality of files.
3. The method of claim 1 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
4. The method of claim 1 , wherein the second plurality of segments already deduplicated correspond to a plurality of filemaps.
5. The method of claim 1 , wherein the second plurality of segments already deduplicated correspond to a plurality of datastore suitcases.
6. The method of claim 5 , wherein a datastore suitcase in the plurality of datastore suitcases further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
7. The method of claim 1 , wherein segment IDs are evaluated using a segment ID database at the host.
8. The method of claim 1 , wherein modifying references and reference counts at the target system for the first plurality of segments and a second plurality of segments comprises.
9. An apparatus, comprising:
an input interface configured to receive a request to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network, wherein the host communicates over the network with the target system by using a file level protocol, wherein the target system is a file server;
a processor configured to determine at the host whether segments in the plurality of files have already been deduplicated at the target system in a deduplication system by comparing segment identifiers (segment IDs) for the segments to match segment IDs for segments already deduplicated, wherein segment IDs are evaluated by transmitting segment IDs to the target system to compare the segment IDs with entries in a deduplication dictionary;
an output interface configured to transmit a first plurality of segments over a network to the target system by using the file level protocol, the first plurality of segments not already deduplicated at the target system, and not transmitting a second plurality of segments to the target system to reduce network bandwidth utilization, the second plurality of segments already deduplicated at the target system;
wherein references and reference counts at the target system are modified for the first plurality of segments and the second plurality of segments, wherein modifying references and reference counts includes modifying a filemap, dictionaries, and a datastore suitcase for both already deduplicated and not already deduplicated segments;
wherein the dictionary is used to identify duplicate data segments and point to the location of the data segments, the datastore suitcase includes an index portion and a data portion to allow a system to perform a bulk read of the index portion to obtain offset data to allow parallel reads of large amounts of data in the data section and further includes three offset, reference count pairs which map to the data segments of the filemap file.
10. The apparatus of claim 9 , wherein the plurality of segments span a plurality of files.
11. The apparatus of claim 9 , wherein the plurality of files are determined to be container or non-container files based on the file types associated with the files.
12. The apparatus of claim 9 , wherein the second plurality of segments already deduplicated correspond to a plurality of filemaps.
13. The apparatus of claim 9 , wherein the second plurality of segments already deduplicated correspond to a plurality of datastore suitcases.
14. The apparatus of claim 13 , wherein a datastore suitcase in the plurality of datastore suitcases further comprises a plurality of reference counts corresponding to the plurality of deduplicated data segments.
15. The apparatus of claim 9 , wherein segment IDs are evaluated using a segment ID database at the host.
16. A non-transitory computer readable medium, comprising:
computer code for receiving a request at a host to deduplicate a plurality of files, the host and a plurality of other hosts connected to a target system and a plurality of other target systems over a network, wherein the host communicates over the network with the target system by using a file level protocol, wherein the target system is a file server;
computer code for determining at the host whether segments in the plurality of files have already been deduplicated at the target system in a deduplication system by comparing segment identifiers (segment IDs) for the segments to match segment IDs for segments already deduplicated, wherein segment IDs are evaluated by transmitting segment IDs to the target system to compare the segment IDs with entries in a deduplication dictionary;
computer code for transmitting a first plurality of segments over a network to the target system by using the file level protocol, the first plurality of segments not already deduplicated at the target system, and not transmitting a second plurality of segments to the target system to reduce network bandwidth utilization, the second plurality of segments already deduplicated at the target system;
computer code for modifying references and reference counts at the target system for the first plurality of segments and the second plurality of segments, wherein modifying references and reference counts includes modifying a filemap, dictionaries, and a datastore suitcase for both already deduplicated and not already deduplicated segments;
wherein the dictionary is used to identify duplicate data segments and point to the location of the data segments, the datastore suitcase includes an index portion and a data portion to allow a system to perform a bulk read of the index portion to obtain offset data to allow parallel reads of large amounts of data in the data section and further includes three offset, reference count pairs which map to the data segments of the filemap file.