Method and system for downloading network data at a controlled data transfer rate
View Patent ↗A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target band width is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.
1. A method of retrieving data over a network at a target bandwidth, B T , comprising:
(1) transmitting a request for data to a server over the network;
(2) receiving the data from the server over the network;
(3) calculating a wait time based on the target bandwidth and an aggregate bytes count, bytes AGG , wherein bytes AGG is an aggregate number of bytes received from the server, and wherein step (3) includes
(A) determining a start time, T START , based on the request for data of step (1),
(B) incrementing the aggregate bytes count, bytes AGG , by the number of bytes received in step (2),
(C) determining a current time, T NOW , and
(D) calculating the wait time according to the equation
wait time=(bytes AGG )/ B T −( T NOW −T START );
(4) waiting the calculated wait time;
(5) transmitting a request for additional data to the server over the network after step (4); and
(6) receiving the additional data from the server over the network.
2. The method of claim 1 , further comprising:
(7) repeating steps (4)-(6) for further additional data until all desired data is retrieved from the server over the network.
3. The method of claim 1 , wherein step (3) further includes
(E) repeating steps (1) and (2) a plurality of times for additional data;
wherein step (B) includes incrementing the aggregate bytes count, bytes AGG , by the number of bytes received in each step (2).
4. A computer system, comprising:
transmitting means for transmitting requests for data to a server over a network;
receiving means for receiving said data from said server over said network; and
a timing module that calculates a wait time based on an aggregate bytes count, bytes AGG , and a target bandwidth, B T , at which rate data is desired to be retrieved from said server over said network, wherein bytes AGG is an aggregate number of bytes received from said server;
wherein said transmitting means delays transmitting requests for data to said server over said network by said calculated wait time;
wherein said timing module determines a start time, T START , corresponding to when said transmitting means transmits a request for data to said server over said network;
wherein said timing module detects a number of bytes received by said receiving means due to a transmitted request;
wherein said timing module increments said aggregate bytes count, bytes AGG , by said number of bytes received;
wherein said timing module determines a current time, T NOW , after at least one iteration of said timing module detecting a number of bytes received by said receiving means due to a transmitted request; and
wherein said timing module calculates said wait time according to the equation
wait time=(bytes AGG )/ B T −( T NOW −T START ).
5. The computer system of claim 4 , wherein said network is the Internet.
6. A computer program product comprising a tangible computer usable storage medium having computer program logic recorded thereon for enabling a processor to retrieve data over a network at a target bandwidth, B T , comprising:
calculating means for enabling a processor to calculate a wait time based on an aggregate bytes count, bytes AGG , and a target bandwidth, B T , at which rate data is desired to be retrieved from a server over said network, wherein bytes AGG is an aggregate number of bytes received from said server; and
delaying means for enabling a processor to delay transmitting requests for data to said server over said network by said calculating wait time;
determining means for enabling a processor to determine start time, T START , corresponding to when a request for data is transmitted to said server over said network;
detecting means for enabling a processor to detect a number of bytes received due to a transmitted request;
incrementing means for enabling a processor to increment said aggregate bytes count, bytes AGG , said number of bytes received; and
determining means for enabling a processor to determine a current time, T NOW , after at least one iteration of detecting a number of bytes received due to a transmitted request;
wherein calculating means enables a processor to calculate said wait time according to the equation
wait time=(bytes AGG )/ B T −( T NOW −T START ).
7. The computer program product of claim 6 , wherein the network is the Internet.