Systems and methods for replication time estimation in a data deduplication system
Systems and methods for of determining a replication time in a deduplicated file system are disclosed. Maximum streams are determined based on a number of allocated streams on a source node and a number of allocated streams on a target node. An available network bandwidth between the source node and the target node is determined. A delta time is estimated based at least on one or more duplicate fingerprints between a logical space unit of the source node and the target node by using at least one source smart filter and at least one target smart filter. The replication time is determined based on the maximum streams, the available network bandwidth between the source and target nodes, the estimated delta time, and a number of unique fingerprints that exist between the logical space unit of the source node and the target node.
1. A computer-implemented method of determining a replication time in a deduplicated file system, the method comprising:
determining maximum streams based on a number of allocated streams on a source node and a number of allocated streams on a target node;
determining an available network bandwidth between the source node and the target node;
estimating a delta time based at least on one or more duplicate fingerprints between a logical space unit of the source node and the target node by using at least one source smart filter and at least one target smart filter; and
determining the replication time based on the maximum streams, the available network bandwidth between the source and target nodes, the estimated delta time, and a number of unique fingerprints that exist between the logical space unit of the source node and the target node.
2. The method of claim 1 , wherein determining the available network bandwidth between the source node and the target node is based on a throughput between the source and target nodes.
3. The method of claim 1 , wherein estimating the delta time is further based on a time needed to send/receive a smallest batch of fingerprints.
4. The method of claim 1 , further comprising: determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node based on (i) a cardinality of unique fingerprints in the logical space unit of the source node and (ii) a cardinality of a union of all logical space units of the target node.
5. The method of claim 4 , wherein determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node comprises determining the cardinality of the unique fingerprints in the logical space unit of the source node based on (i) an intersection cardinality of the logical space unit and deleted fingerprints by garbage collection and (ii) a cardinality of the logical space unit.
6. The method of claim 1 , wherein determining the maximum streams comprises determining the maximum streams using a minimum function of the number of allocated streams on the source node and the number of allocated streams on the target node.
7. The method of claim 1 , wherein the at least one source smart filter and the at least one target smart filter are implemented using HyperLogLog (HLL) or HLL++.
8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:
determining maximum streams based on a number of allocated streams on a source node and a number of allocated streams on a target node;
determining an available network bandwidth between the source node and the target node;
estimating a delta time based at least on one or more duplicate fingerprints between a logical space unit of the source node and the target node by using at least one source smart filter and at least one target smart filter; and
determining a replication time based on the maximum streams, the available network bandwidth between the source and target nodes, the estimated delta time, and a number of unique fingerprints that exist between the logical space unit of the source node and the target node.
9. The non-transitory machine-readable medium of claim 8 , wherein determining the available network bandwidth between the source node and the target node is based on a throughput between the source and target nodes.
10. The non-transitory machine-readable medium of claim 8 , wherein estimating the delta time is further based on a time needed to send/receive a smallest batch of fingerprints.
11. The non-transitory machine-readable medium of claim 8 , wherein the operations further comprise: determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node based on (i) a cardinality of unique fingerprints in the logical space unit of the source node and (ii) a cardinality of a union of all logical space units of the target node.
12. The non-transitory machine-readable medium of claim 11 , wherein determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node comprises determining the cardinality of the unique fingerprints in the logical space unit of the source node based on (i) an intersection cardinality of the logical space unit and deleted fingerprints by garbage collection and (ii) a cardinality of the logical space unit.
13. The non-transitory machine-readable medium of claim 8 , wherein determining the maximum streams comprises determining the maximum streams using a minimum function of the number of allocated streams on the source node and the number of allocated streams on the target node.
14. The non-transitory machine-readable medium of claim 8 , wherein the at least one source smart filter and the at least one target smart filter are implemented using HyperLogLog (HLL) or HLL++.
15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including:
determining maximum streams based on a number of allocated streams on a source node and a number of allocated streams on a target node;
determining an available network bandwidth between the source node and the target node;
estimating a delta time based at least on one or more duplicate fingerprints between a logical space unit of the source node and the target node by using at least one source smart filter and at least one target smart filter; and
determining a replication time based on the maximum streams, the available network bandwidth between the source and target nodes, the estimated delta time, and a number of unique fingerprints that exist between the logical space unit of the source node and the target node.
16. The data processing system of claim 15 , wherein determining the available network bandwidth between the source node and the target node is based on a throughput between the source and target nodes.
17. The data processing system of claim 15 , wherein estimating the delta time is further based on a time needed to send/receive a smallest batch of fingerprints.
18. The data processing system of claim 15 , wherein the operations further include: determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node based on (i) a cardinality of unique fingerprints in the logical space unit of the source node and (ii) a cardinality of a union of all logical space units of the target node.
19. The data processing system of claim 18 , wherein determining the number of unique fingerprints that exist between the logical space unit of the source node and the target node comprises determining the cardinality of the unique fingerprints in the logical space unit of the source node based on (i) an intersection cardinality of the logical space unit and deleted fingerprints by garbage collection and (ii) a cardinality of the logical space unit.
20. The data processing system of claim 15 , wherein the at least one source smart filter and the at least one target smart filter are implemented using HyperLogLog (HLL) or HLL++.