Adaptive data transmission
Methods, program products, and systems implementing adaptive data transmission are disclosed. A universal data collector can act as an intermediary between a log producer and a log processor. The universal data collector can be connected to the log producer through a first, reliable network. The universal data collector can be connected to the log processor through a second, less reliable network. The universal data collector can perform a multi-phase commit of data to ensure that the log data are saved by the log processor. The universal data collector can smooth the data transmission from the universal data collector to the log processor through a network, including adaptively buffering the log data according to available network bandwidth of the network.
1. A method, comprising:
receiving, by a data collector comprising one or more computer processors, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector;
subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion of the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including:
a first threshold on bandwidth usage and a second threshold on log processor response time; and
rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and
adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including:
upon determining, by a collection manager comprising one or more computer processors, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and
upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector.
2. The method of claim 1 , wherein:
receiving log data from the one or more log producers includes receiving log data from one or more log producing devices through a local area network (LAN); and
the communications network between the data collector and the log processor includes a wide area network (WAN).
3. The method of claim 1 , wherein:
the data collector organizes the received log data into units and enters the units into a queue stored in the buffer;
transmitting the stored portion of the log data to the log processor includes transmitting units of log data stored in the queue sequentially; and
the data collector removes a first unit of log data from the queue upon receiving an acknowledgement from the log processor that the first unit of log data has been processed.
4. The method of claim 3 , wherein transmitting the stored portion of the log data to the log processor includes retransmitting a second unit of log data upon determining that, after a specified threshold length of time, an acknowledgement that the second unit of log data has been processed has not been received.
5. The method of claim 1 , wherein adapting the data collector for transmitting the stored portion of the log data to the log processor includes storing a first portion of the stored log data on the data collector such that a second portion of the stored log data is transmitted to the log processor using available network bandwidth.
6. The method of claim 1 , wherein increasing the size of the buffer causes a reduction in the bandwidth utilization rate by reducing transmission of the first portion of the log data and increasing storage of the second portion of the log data.
7. A non-transitory storage device storing a computer program product, the computer program product, when executed, operable to cause a data collector and a collection manager each comprising one or more processors to perform operations comprising:
receiving, by the data collector, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector;
subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including:
a first threshold on bandwidth usage and a second threshold on log processor response time; and
rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and
adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including:
upon determining, by the collection manager, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and
upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector.
8. The non-transitory storage device of claim 7 , wherein:
receiving log data from the one or more log producers includes receiving log data from one or more log producing devices through a local area network (LAN); and
the communications network between the data collector and the log processor includes a wide area network (WAN).
9. The non-transitory storage device of claim 7 , wherein:
the data collector organizes the received log data into units and enters the units into a queue stored in the buffer;
transmitting the stored portion of the log data to the log processor includes transmitting units of log data stored in the queue sequentially; and
the data collector removes a first unit of log data from the queue upon receiving an acknowledgement from the log processor that the first unit of log data has been processed.
10. The non-transitory storage device of claim 9 , wherein transmitting the stored portion log data to the log processor includes retransmitting a second unit of log data upon determining that, after a specified threshold length of time, an acknowledgement that the second unit of log data has been processed has not been received.
11. The non-transitory storage device of claim 7 , wherein adapting the data collector for transmitting the stored portion of the log data to the log processor includes storing a first portion of the stored log data on the data collector such that a second portion of the stored log data is transmitted to the log processor using available network bandwidth.
12. The non-transitory storage device of claim 11 , wherein transmitting the stored portion of the log data to the log processor includes transmitting the stored portion of the log data before receiving an acknowledgement.
13. The non-transitory storage device of claim 11 , the operations further comprising transmitting to the log processor an instruction instructing the log processor to send the acknowledgement to the data collector after a specified quantity of log data have been successfully stored in the buffer.
14. The non-transitory storage device of claim 7 , wherein increasing the size of the buffer causes a reduction in the bandwidth utilization rate by reducing transmission of the first portion of the log data and increasing storage of the second portion of the log data.
15. A system comprising:
a data collector and a collection manager each comprising one or more computer processors; and
a non-transitory storage device storing a computer program product, the computer program product, when executed, operable to cause the data collector and collection manager to perform operations comprising:
receiving, by the data collector, log data from one or more log producers to be transmitted to a log processor through a communications network, the log processor being configured to provide an acknowledgement to the data collector for each unit of log data received from the data collector;
subject to a transmission constraint of transmitting the received log data to the log processor, transmitting, by the data collector, a first portion of the received log data to the log processor and storing, by the data collector, a second portion the received log data into a buffer configured to adapt to the transmission constraint, the transmission constraint including:
a first threshold on bandwidth usage and a second threshold on log processor response time; and
rules on adapting a size of the buffer or computing resources allocated to the log processor when the thresholds are reached; and
adapting the data collector or the log processor for transmitting the stored portion of the log data to the log processor in accordance with the rules in the transmission constraint, including:
upon determining, by the collection manager, that a bandwidth utilization rate of the communication network reached the first threshold, instructing, by the collection manager, the data collector to adapt to the bandwidth utilization rate by increasing the size of the buffer, increasing the size of the buffer including allocating additional storage space for the buffer wherein, when the bandwidth utilization rate no longer reaches the first threshold, the additional storage space is reallocated for other uses, and wherein determining that the bandwidth utilization rate reached the first threshold comprises determining, by the collection manager, that traffic of transmitting the first portion of the received log data from the data collector to the log processor has been utilizing full available bandwidth of the communication network for a given period of time; and
upon determining, by the collection manager, that a promptness for providing the acknowledgement by the log processor satisfies the second threshold, configuring the log processor to allocate more computing resources to process log data transmitted from the data collector.
16. The system of claim 15 , wherein:
receiving log data from the one or more log producers includes receiving log data from one or more log producing devices through a local area network (LAN); and
the communications network between the data collector and the log processor includes a wide area network (WAN).
17. The system of claim 15 , wherein:
the data collector organizes the received log data into units and enters the units into a queue stored in the buffer;
transmitting the stored portion of the log data to the log processor includes transmitting units of log data stored in the queue sequentially; and
the data collector removes a first unit of log data from the queue upon receiving an acknowledgement from the log processor that the first unit of log data has been processed.
18. The system of claim 17 , wherein transmitting the stored portion of the log data to the log processor includes retransmitting a second unit of log data upon determining that, after a specified threshold length of time, an acknowledgement that the second unit of log data has been processed has not been received.
19. The system of claim 15 , the operations further comprising:
determining a first feature indicator, the first feature indicator indicating that one or more first features of processing the log data are available on the data collector;
receiving a second feature indicator from the log processor, the second feature indicator indicating that one or more second features of processing the log data are available on the log processor;
determining that at least one first feature does not correspond to the one or more second features; and
configuring the data collector such that the first feature that does not correspond to the one or more second features is not applied in processing the log data to be transmitted to the log processor.
20. The system of claim 15 , wherein increasing the size of the buffer causes a reduction in the bandwidth utilization rate by reducing transmission of the first portion of the log data and increasing storage of the second portion of the log data.