Data replication with dynamic compression
View Patent ↗A method for replicating data between two or more network connected data storage devices, the method including dynamically determining whether to compress data prior to transmitting across the network based, at least in part, on bandwidth throughput between the network connected data storage devices. If it has been determined to compress the data, the method involves compressing the data and transmitting the compressed data over the network. If it has been determined not to compress the data, the method involves transmitting the data, uncompressed, over the network. Dynamically determining whether to compress data may include comparing bandwidth measurements with a predetermined policy defining when compression should be utilized. In some embodiments, the policy may define that compression should be utilized when an estimated time for compressing the data and transmitting the compressed data is less than an estimated time for transmitting the data uncompressed.
1. A method for dynamically determining whether to compress data prior to transmitting across a network from a first network connected data storage device to a second network connected data storage device, the method comprising:
periodically transmitting sample data from the first network connected data storage device to the second network connected data storage device to obtain a measurement indicative of bandwidth throughput between the first and second network connected data storage devices;
comparing the measurement indicative of bandwidth throughput with a predetermined policy defining when compression should be utilized; and
based on the comparison and CPU resources available for data compression, dynamically determining whether to compress the data prior to transmitting from the first network connected data storage device to the second network connected data storage device;
wherein if compression of the data would be below a predetermined threshold, determining not to compress the data prior to transmitting across the network from the first network connected data storage device to the second network connected data storage device regardless of the comparison and CPU resources available.
2. A method for dynamically determining whether to compress data prior to transmitting across a network from a first network connected data storage device to a second network connected data storage device, the method comprising:
during transmission of a first data from the first network connected data storage device to the second network connected data storage device, compressing a second data;
if compression of the second data has not completed by the time transmission of the first data from the first network connected data storage device to the second network connected data storage device has completed, then transmitting the second data, uncompressed, from the first network connected data storage device to the second network connected data storage device; and
wherein if a total amount of compression in the size of the second data would be below a predetermined threshold, it is determined not to compress the second data prior to transmitting across the network from the first network connected data storage device to the second network connected data storage device.
3. The method of claim 1 , wherein if the comparison of the measurement indicative of bandwidth throughput with the predetermined policy indicates that compression should be utilized, unless compression of the data would be below a predetermined threshold, compressing all data sent between the first network connected data storage device and the second network connected data storage device until a subsequent periodic sample data transmission and comparison to the policy are made.
4. The method of claim 1 , wherein the step of transmitting sample data to obtain a measurement of bandwidth throughput is repeated after a predetermined time interval.
5. The method of claim 1 , wherein the step of transmitting sample data to obtain a measurement of bandwidth throughput is repeated after a predetermined amount of data has been transmitted from the first network connected data storage device to the second network connected data storage device.
6. The method of claim 1 , wherein the policy defines that compression should be utilized when an estimated time for compressing the data and transmitting the compressed data is less than an estimated time for transmitting the data uncompressed.
7. The method of claim 1 , wherein the sample data comprises a first portion of the data to be transmitted from the first network connected data storage device to the second network connected data storage device.
8. The method of claim 1 , wherein the policy further defines that compression should be utilized for a first portion of the data and not for a second portion of the data so as to stagger the compression for transmission of the data to the second network connected data storage device.