Server load management for data migration
A system and method for controlling a migration of data items in a data directory from a source system to a destination system by setting one or more target properties for implementing the migration are disclosed. A migration controller can be employed to obtain statistical information regarding the migration. The statistical information can then be used to compute the one or more target properties. For example, a target number of network connections for achieving an optimal total throughput and concurrency can be computed based on an average throughput per network connection. The target number of network connections may be determined by a migration monitor, which may set the determined target number of network connections as a policy to be enforced by the migration controller.
1. A method for optimizing performance of data transmission between a source server and a destination server, the method comprising:
determining a set of average throughputs per network connection between the source server and the destination server, each of the average throughputs in the set corresponding to a number of network connections between the source server and the destination server;
identifying a specific number of network connections corresponding to a specific change in the average throughput per connection based on a change in the number of network connections between the source server and the destination server;
setting the specific number of network connections as a target number of network connections; and
implementing the data transmission between the source server and the destination system using the target number of network connections.
2. The method of claim 1 , wherein the specific number of network connections is identified based on a decrease in average throughput per connection between the specific number of network connects and specific number plus one of connections.
3. The method of claim 1 , wherein the amount of the data requested to be processed for each network connection is the same.
4. The method of claim 1 , wherein identifying the specific number of network connections comprises:
plotting a series of points in a Cartesian coordinate system, wherein the X coordinate of each of the series of the points corresponds to the number of network connections corresponding to each of the average throughputs in the set and the Y coordinate of each of the series of the points corresponds to the average throughputs per network connection in the set;
connecting the series of points to form a curve; and
identifying the specific number of network connections using the curve.
5. The method of claim 1 , wherein identifying the specific number of network connections comprises:
determining a function of average throughput per network connection with respect to a quantity of network connections provided by the source server; and
fitting the set of average throughputs against the number of network connections corresponding to each of the average throughputs in the set to obtain variables of the function.
6. A system for optimizing performance of a source server configured to facilitate data transmission between the source server and a destination server, the system comprising one or more processors configured to perform:
determining a set of average throughputs per network connection between the source server and the destination server, each of the average throughputs in the set corresponding to a number of network connections between the source server and the destination server;
identifying a specific number of network connections corresponding to a specific change in the average throughput per connection based on a change in the number of network connections between the source server and the destination server;
setting the specific number of network connections as a target number of network connections; and
implementing the data transmission between the source server and the destination system using the target number of network connections.
7. The system of claim 6 , wherein the specific number of network connections is identified based on a decrease in average throughput per connection between the specific number of network connects and specific number plus one of connections.
8. The system of claim 6 , wherein the amount of the data requested to be processed for each network connection is the same.
9. The system of claim 6 , wherein identifying the target number of network connections comprises:
plotting a series of points in a Cartesian coordinate system, wherein the X coordinate of each of the series of the points corresponds to the number of network connections corresponding to each of the average throughputs in the set and the Y coordinate of each of the series of the points corresponds to the average throughputs per network connection in the set;
connecting the series of points to form a curve; and
identifying the specific number of network connections using the curve.
10. The method of claim 6 , wherein identifying the target number of network connections comprises:
determining a function of average throughput per network connection with respect to a quantity of network connections between the source server and destination server; and
fitting the set of average throughputs against the number of network connections corresponding to each of the average throughputs in the set to obtain variables of the function.