Efficient resource utilization for cross-site deduplication
Methods and apparatuses are described to provide a mechanism to improve replication performance by understanding available bandwidths of multiple source servers and load balancing the replication among the source servers based on fingerprints of data units to be replicated. A separate source server can be identified for one source server to replicate the data units to a target server. A replication request is sent from the one source server to the separate source server to identify the data units. A reply may be received from the separate source server to indicate that one or more of the data units are replicated from the separate source server. Subsequently, the remaining ones of the data units not indicated in the reply may be sent directly from the one source server to the target server without sending the one or more data units indicated in the reply.
1. A machine-readable non-transitory storage medium having instructions therein, which when executed by a machine, cause the machine to perform operations comprising:
identifying, at a first source server, one or more data units to be replicated to a target server, the one or more data units being stored at the first source server;
identifying, at the first source server, a second source server for replicating at least a subset of the one or more data units from the second source server to the target server, the subset of the one or more data units to be replicated from the second source server being stored at both the first source server and the second source server, the second source server coupled with the target server, the second source sever capable of performing data replication to the target server;
sending a replication request from the first source server to the second source server, the replication request identifying the one or more data units for replication;
receiving a reply to the replication request from the second source server, the reply indicating the subset of the one or more data units are replicated from the second source server, wherein remaining ones of the one or more data units are not replicated from the second source server for the replication request; and
sending, subsequent to receiving the reply, at least one of the remaining ones of the one or more data units from the first source server to the target server for the replication of the one or more data units without sending the subset of the one or more data units indicated in the reply,
wherein replication of a portion of the subset of the one or more data units from the second source server to the target server takes place concurrently with replication of a portion of the remaining ones of the one or more data units from the first source server to the target server for the replication of the one or more data units.
2. The machine-readable non-transitory storage medium of claim 1 , wherein a plurality of source servers are separately coupled to the target server, the source servers including the second source server, each source server capable of performing the data replication to the target server, wherein the identification comprises:
maintaining a topology map dynamically during runtime, the topology map identifying the source servers, wherein the identification of the second source server is based on the topology map.
3. The machine-readable non-transitory storage medium of claim 2 , wherein the source servers include a master server monitoring runtime states of the source servers and wherein the topology map is received from the master server.
4. The machine-readable non-transitory storage medium of claim 2 , the identification further comprising:
selecting one or more eligible source servers for the replication of the one or more data units from the plurality of source servers based on the topology map and wherein the selected one or more eligible source servers include the second source server.
5. The machine-readable non-transitory storage medium of claim 4 , wherein the topology map indicates whether a source server is currently performing the data replication to the target server, and wherein an eligible source server is not currently performing the data replication to the target server.
6. The machine-readable non-transitory storage medium of claim 4 , wherein the topology map indicates a connection bandwidth associated with each source server with the target server, and wherein the eligible source servers are selected according to the connection bandwidth associated with each source server.
7. The machine-readable non-transitory storage medium of claim 6 , wherein one source server is not selected if the one source server is associated with a connection bandwidth smaller than a threshold value.
8. The machine-readable non-transitory storage medium of claim 7 , wherein the first source server is coupled with the target server via a network connection having a particular connection bandwidth and wherein the threshold value is determined based on the particular connection bandwidth.
9. The machine-readable non-transitory storage medium of claim 4 , the operations further comprising:
determining an ordering relationship among the eligible source servers according to amounts of network bandwidth associated with the eligible source servers to the target server, wherein the second source server is associated with a highest amount of network bandwidth among the eligible source servers according to the ordering relationship.
10. The machine-readable non-transitory storage medium of claim 9 , wherein a third source server is ordered next to the second source server among the eligible source servers according to the ordering relationship, the operations further comprising:
sending, in response to the reply received from the second source server, an updated replication request from the first source server to the third source server, wherein the updated replication request identifying the remaining ones of the one or more data units for replication, the updated replication request without identifying the subset of the one or more data units indicated in the reply from the second source server; and
receiving a separate reply from the third source server, the separate reply indicating one or more of the remaining ones of the one or more data units are replicated from the third source server, wherein the one or more of the remaining ones of the one or more data units do not include the at least one of the remaining ones of the one or more data units sent from the first source server to the target server.
11. The machine-readable non-transitory storage medium of claim 4 , wherein the identification further comprises:
generating a fingerprint describing the one or more data units to be replicated to the target server;
sending inquiry messages to the eligible source servers, each inquiry message identifying the one or more data units for replication; and
collecting response messages from the eligible source servers in response to the inquiry messages, each response message received from one of the eligible source servers, the response message indicating which of the one or more data units are available in the one eligible source server, wherein a particular response message received from the second source server indicates the subset of the one and more data units are available in the second source server.
12. The machine-readable non-transitory storage medium of claim 11 , further comprising:
identifying one or more of the eligible source servers to replicate separate portions of the one or more data units, wherein the identified eligible source servers include the second source server to replicate one of the separate portions of the one or more data units includes the subset of the one or more data units.
13. The machine-readable non-transitory storage medium of claim 12 , further comprising:
sending multiple replication requests to the identified source servers to separately replicate the separate portions of the one or more data units, the multiple replication requests including the replication request to the second source server.
14. The machine-readable non-transitory storage medium of claim 13 , wherein the multiple replication requests are sent concurrently.
15. The machine-readable non-transitory storage medium of claim 13 , further comprising:
receiving multiple replies in response to the multiple replication requests from the identified source servers, the multiple replies including the reply from the second source server; and
determining which of the one or more data units to be replicated from the first source server, wherein the at least one of the remaining ones of the one or more data units are determined to be replicated from the first source server.
16. The machine-readable non-transitory storage medium of claim 1 , wherein the one or more data units are stored in a storage component coupled with the first source server and wherein each data unit represents a fixed sized partition of the storage component.
17. A computer implemented method comprising:
identifying, at a first source server, one or more data units to be replicated to a target server, the one or more data units being stored at the first source server;
identifying, at the first source server, a second source server for replicating at least a subset of one or more data units from the second source server to the target server, the subset of the one or more data units to be replicated from the second source server being stored at both the first source server and the second source server, the second source server coupled with the target server, the second site sever performing data replication to the target server;
sending a replication request from the first source server to the second source server, the replication request identifying the one or more data units for replication;
receiving a reply to the replication request from the second source server, the reply indicating the subset of the one or more data units are replicated from the second source server, wherein remaining ones of the one or more data units are not replicated from the second source server for the replication request; and
sending, subsequent to receiving the reply, at least one of the remaining ones of the one or more data units from the first source server to the target server for the replication of the one or more data units without sending the subset of the one or more data units indicated in the reply,
wherein replication of a portion of the subset of the one or more data units from the second source server to the target server takes place concurrently with replication of a portion of the remaining ones of the one or more data units from the first source server to the target server for the replication of the one or more data units.
18. The computer implemented method of claim 17 , wherein a plurality of source servers are separately coupled to the target server, the source servers including the second source server, each source server performing the data replication to the target server, wherein the identification comprises:
maintaining a topology map dynamically during runtime, the topology map identifying the source servers, wherein the identification of the second source server is based on the topology map.
19. The computer implemented method of claim 18 , the identification further comprising:
selecting one or more eligible source servers for the replication of the one or more data units from the plurality of source servers based on the topology map and wherein the selected one or more eligible source servers include the second source server.
20. The computer implemented method of claim 19 , the operations further comprising:
determining an ordering relationship among the eligible source servers according to amounts of network bandwidth associated with the eligible source servers to the target server, wherein the second source server is associated with a highest amount of network bandwidth among the eligible source servers according to the ordering relationship.
21. The computer implemented method of claim 20 , wherein a third source server is ordered next to the second source server among the eligible source servers according to the ordering relationship, the operations further comprising:
sending, in response to the reply received from the second source server, an updated replication request from the first source server to the third source server, wherein the updated replication request identifying the remaining ones of the one or more data units for replication, the updated replication request without identifying the subset of the one or more data units indicated in the reply from the second source server; and
receiving a separate reply from the third source server, the separate reply indicating one or more of the remaining ones of the one or more data units are replicated from the third source server, wherein the one or more of the remaining ones of the one or more data units do not include the at least one of the remaining ones of the one or more data units sent from the first source server to the target server.
22. The computer implemented method of claim 19 , wherein the identification further comprises:
generating a fingerprint describing the one or more data units to be replicated to the target server;
sending inquiry messages to the eligible source servers, each inquiry message identifying the one or more data units for replication; and
collecting response messages from the eligible source servers in response to the inquiry messages, each response message received from one of the eligible source servers, the response message indicating which of the one or more data units are available in the one eligible source server, wherein a particular response message received from the second source server indicates the subset of the one and more data units are available in the second source server.
23. A computer system comprising:
a memory storing executable instructions;
a network interface coupled to one or more source servers and a target server in a cross-site configuration;
a storage device storing data units;
a processor coupled to the memory, the network interface and the storage device, the processor to execute the executable instructions from the memory to
identify one or more data units to be replicated to a target server, the one or more data units being stored at the computer system;
identify a separate source server in the cross-site configuration for replicating a subset of the one or more data units from the separate source server to a target server, the subset of the one or more data units to be replicated from the separate source server being stored at both the computer system and the separate source server, the separate source capable of performing data replication to the target server,
send a replication request to the separate source server, the replication request identifying the one or more data units for replication,
receive a reply to the replication request from the separate source server, the reply indicating the subset of the one or more data units are replicated from the separate source server, wherein remaining ones of the one or more data units are not replicated from the separate source server for the replication request, and
send, subsequent to receiving the reply, at least one of the remaining ones of the one or more data units to the target server for the replication of the one or more data units without sending the subset of the one or more data units indicated in the reply,
wherein replication of a portion of the subset of the one or more data units from the second source server to the target server takes place concurrently with replication of a portion of the remaining ones of the one or more data units from the first source server to the target server for the replication of the one or more data units.