Bulk network transmissions using multiple connections primed to optimize transfer parameters
View Patent ↗A system for bulk network transmissions using multiple connections primed to optimize transfer parameters includes one or more processors and memory coupled to the processors. The memory stores program instructions executable by the processors to establish a plurality of network connections designated to be used for a single bulk data transfer. The instructions are further executable to adjust the values of one or more configuration parameters of each of the connections. The adjusting includes sending one or more priming packets over each of the connections. The instructions are also executable to perform the bulk transfer as a sequence of sub-transfers. Each sub-transfer includes a transmission of a predetermined amount of application data over each connection of a selected subset of the plurality of network connections.
1. A system, comprising:
one or more processors; and
memory coupled to the one or more processors, wherein the memory comprises program instructions executable by the one or more processors to:
establish a plurality of network connections designated to be used for a bulk transfer;
adjust one or more configuration parameters associated with each network connection of the plurality of network connections, wherein to adjust the one or more configuration parameters, the instructions are further executable to send one or more priming packets associated with the bulk data transfer over each network connection to generate a plurality of primed network connections;
perform the bulk transfer as a sequence of sub-transfers, wherein each sub-transfer includes a transmission of a predetermined amount of the data over each primed network connection of a selected subset of the plurality of primed network connections, wherein a reserved subset of the plurality of primed network connections are not initially used in performing the bulk transfer as the sequence of sub-transfers;
determine whether data transmission over each respective network connection selected for a corresponding sub-transfer of the sequence of sub-transfers meets a threshold performance criterion; and
in response to determining that the data transmission over a respective one of the network connections does not meet the threshold performance criterion, automatically determine that the respective one of the network connections is an unsatisfactory connection and automatically retransmit at least a portion of the corresponding sub-transfer over one of the primed network connections of the reserved subset of the plurality of primed network connections.
2. The system as recited in claim 1 , wherein the one or more configuration parameters associated with a network connection of the plurality of network connections include a parameter indicative of an allowed amount of unacknowledged data on the network connection.
3. The system as recited in claim 1 , wherein the one or more configuration parameters associated with a network connection of the plurality of network connections include a transmit window size.
4. The system as recited in claim 1 , wherein to adjust the one or more configuration parameters, the instructions are further executable to:
determine, after sending a particular priming packet of the plurality of priming packets, whether a particular configuration parameter of the one or more configuration parameters has reached a predetermined setting;
if the particular configuration parameter has not reached the predetermined setting, send an additional priming packet over the network connection; and
if the particular configuration parameter has reached the predetermined setting, determine that no additional priming packets are to be sent over the network connection.
5. The system as recited in claim 1 , wherein the sequence of sub-transfers includes a particular sub-transfer followed by another sub-transfer, wherein the subset of primed network connections selected for the particular sub-transfer differs from the subset of primed network connections selected for the another data transfer.
6. The system as recited in claim 1 , wherein the instructions are further executable to:
in response to determining that the data transmission over the respective one of the network connections does not meet the threshold performance criterion,
close the respective one of the network connections;
open an additional network connection;
adjust one or more configuration parameters associated with the additional network connection, wherein said adjusting comprises sending one or more priming packets over the additional network connection; and
designate the additional network connection to be included in a set of network connections to be used to implement a remainder of the bulk transfer.
7. The system as recited in claim 1 , wherein the threshold performance criterion comprises a time taken to transmit a packet from a sender to a receiver over each respective network connection.
8. The system as recited in claim 1 , wherein to determine whether the data transmission over each respective network connection meets the threshold performance criterion, the instructions are further executable to:
insert instrumentation data into a packet transmitted over each respective network connection.
9. The system as recited in claim 8 , wherein said instrumentation data is inserted using a user-mode software module, without modifying a networking software stack in use for each respective network connection.
10. The system as recited in claim 1 , wherein the instructions are further executable to:
partition data corresponding to the bulk transfer into a plurality of stripes of a predetermined transmission stripe size; and
transmit consecutive stripes on each of two or more primed network connections selected for a given sub-transfer of the sequence of sub-transfers.
11. A computer-implemented method, comprising:
establishing a plurality of network connections designated to be used for a bulk transfer;
adjusting one or more configuration parameters associated with each network connection of the plurality of network connections, wherein said adjusting comprises sending one or more priming packets associated with the bulk data transfer over each network connection to generate a plurality of primed network connections;
performing the bulk transfer as a sequence of sub-transfers, wherein each sub-transfer includes a transmission of a predetermined amount of the data over each primed network connection of a selected subset of the plurality of primed network connections, wherein a reserved subset of the plurality of primed network connections are not initially used in performing the bulk transfer as the sequence of sub-transfers;
determining whether data transmission over each respective network connection selected for a corresponding sub-transfer of the sequence of sub-transfers meets a threshold performance criterion; and
in response to determining that the data transmission over a respective one of the network connections does not meet the threshold performance criterion, automatically determining that the respective one of the network connections is an unsatisfactory connection and automatically retransmitting at least a portion of the corresponding sub-transfer over one of the primed network connections of the reserved subset of the plurality of primed network connections.
12. The method as recited in claim 11 , wherein said adjusting comprises:
determining, after sending a particular priming packet of the plurality of priming packets, whether a particular configuration parameter of the one or more configuration parameters has reached a predetermined setting;
if the particular configuration parameter has not reached the predetermined setting, sending an additional priming packet over the network connection; and
if the particular configuration parameter has reached the predetermined setting, terminating said adjusting.
13. The method as recited in claim 11 , wherein the sequence of sub-transfers includes a particular sub-transfer followed by another sub-transfer, wherein the subset of primed network connections selected for the particular sub-transfer differs from the subset of primed network connections selected for the another data transfer.
14. The method as recited in claim 11 , further comprising:
in response to determining that the data transmission over the respective one of the network connections does not meet the threshold performance criterion,
closing the respective one of the network connections;
opening an additional network connection;
adjusting one or more configuration parameters associated with the additional network connection, wherein said adjusting comprises sending one or more priming packets over the additional network connection; and
designating the additional network connection to be included in a set of network connections to be used to implement a remainder of the bulk transfer.
15. The method as recited in claim 11 , wherein said determining whether the data transmission over each respective network connection meets the threshold performance criterion comprises:
inserting instrumentation data into a packet transmitted over each respective network connection.
16. The method as recited in claim 11 , further comprising:
partitioning data corresponding to the bulk transfer into a plurality of stripes of a predetermined transmission stripe size; and
transmitting consecutive stripes on each of two or more primed network connections selected for a given sub-transfer of the sequence of sub-transfers.
17. A storage medium storing program instructions, wherein the instructions are computer executable to:
establish a plurality of network connections designated to be used for a bulk transfer;
adjust one or more configuration parameters associated with each network connection of the plurality of network connections, wherein to adjust the one or more configuration parameters, the instructions are further executable to send one or more priming packets associated with the bulk data transfer over each network connection to generate a plurality of primed network connections;
perform the bulk transfer as a sequence of sub-transfers, wherein each sub-transfer includes a transmission of a predetermined amount of the data over each primed network connection of a selected subset of the plurality of primed network connections, wherein a reserved subset of the plurality of primed network connections are not initially used in performing the bulk transfer as the sequence of sub-transfers;
determine whether data transmission over each respective network connection selected for a corresponding sub-transfer of the sequence of sub-transfers meets a threshold performance criterion; and
in response to determining that the data transmission over a respective one of the network connections does not meet the threshold performance criterion, automatically determine that the respective one of the network connections is an unsatisfactory connection and automatically retransmitting at least a portion of the corresponding sub-transfer over one of the primed network connections of the reserved subset of the plurality of primed network connections.
18. The storage medium as recited in claim 17 , wherein to adjust the one or more configuration parameters, the instructions are further executable to:
determine, after sending a particular priming packet of the plurality of priming packets, whether a particular configuration parameter of the one or more configuration parameters has reached a predetermined setting;
if the particular configuration parameter has not reached the predetermined setting, send an additional priming packet over the network connection; and
if the particular configuration parameter has reached the predetermined setting, determine that no additional priming packets are to be sent over the network connection.
19. The storage medium as recited in claim 17 , wherein the sequence of sub-transfers includes a particular sub-transfer followed by another sub-transfer, wherein the subset of primed network connections selected for the particular sub-transfer differs from the subset of primed network connections selected for the another data transfer.
20. The storage medium as recited in claim 17 , wherein the instructions are further executable to:
in response to determining that the data transmission over the respective one of the network connections does not meet the threshold performance criterion,
close the respective one of the network connections;
open an additional network connection;
adjust one or more configuration parameters associated with the additional network connection, wherein said adjusting comprises sending one or more priming packets over the additional network connection; and
designate the additional network connection to be included in a set of network connections to be used to implement a remainder of the bulk transfer.
21. The storage medium as recited in claim 17 , wherein the instructions are further executable to:
partition data corresponding to the bulk transfer into a plurality of stripes of a predetermined transmission stripe size; and
transmit consecutive stripes on each of two or more primed network connections selected for a given sub-transfer of the sequence of sub-transfers.