Optimal and dynamic selection of compression dictionary size and data buffer coalesce infrastructure for greater data reduction and lowered network bandwidth usage of remote data facility replication systems
Storage arrays in a mirror relationship use dynamic compression dictionary size adjustment and data buffer merging to more efficiently compress data associated with maintenance of consistency of a replicated storage object. A compression dictionary size selector selects a supported compression dictionary size based on the aggregate size of the updates enqueued for transmission in the data buffers, e.g., a compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the enqueued updates. A combiner selects enqueued updates based on the selected compression dictionary size, e.g., aggregate enqueued update size less than or equal to, and as close as possible to, the selected compression dictionary size. The selected updates are coalesced and compressed using the selected compression dictionary size.
1. A method, comprising:
repeatedly, and responsive to replica updates being enqueued for transmission to a mirror:
determining an aggregate size of the replica updates enqueued for transmission;
selecting a compression dictionary size based on the aggregate size of the replica updates enqueued for transmission;
selecting ones of the replica updates enqueued for transmission based on the selected compression dictionary size;
coalescing the selected ones of the replica updates;
compressing the coalesced replica updates using the selected compression dictionary size; and
providing the compressed coalesced replica updates to the mirror.
2. The method of claim 1 comprising selecting the compression dictionary size from a plurality of supported compression dictionary sizes.
3. The method of claim 2 comprising selecting the supported compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the replica updates enqueued for transmission.
4. The method of claim 2 comprising selecting the supported compression dictionary size that is a maximum supported compression dictionary size responsive to one of the supported compression dictionary sizes being greater than or equal to the aggregate size of the replica updates enqueued for transmission.
5. The method of claim 2 comprising selecting the supported compression dictionary size that is a minimum supported compression dictionary size.
6. The method of claim 1 comprising selecting the ones of the replica updates enqueued for transmission such that aggregate size of the selected ones of the replica updates is less than or equal to, and as close as possible to, the selected compression dictionary size.
7. The method of claim 1 comprising determining the aggregate size of the replica updates enqueued for transmission based on number of populated data buffers and data buffer size.
8. An apparatus, comprising:
at least one compute node configured to maintain a replicated storage object, the compute node comprising:
data buffers configured to store enqueued updates to the replicated storage object;
a compression dictionary size selector configured to select a compression dictionary size based on aggregate size of the updates enqueued for transmission in the data buffers;
a combiner configured to select ones of the updates enqueued for transmission based on the selected compression dictionary size and coalesce the selected ones of the updates; and
a data reduction module configured to compress the coalesced updates using the selected compression dictionary size;
wherein the compressed coalesced updates are provided to a mirror.
9. The apparatus of claim 8 wherein the compression dictionary size selector is configured to select the compression dictionary size from a plurality of supported compression dictionary sizes.
10. The apparatus of claim 9 wherein the compression dictionary size selector is configured to select the supported compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the replica updates enqueued for transmission.
11. The apparatus of claim 9 wherein the compression dictionary size selector is configured to select the supported compression dictionary size that is a maximum supported compression dictionary size responsive to one of the supported compression dictionary sizes being greater than or equal to the aggregate size of the replica updates enqueued for transmission.
12. The apparatus of claim 9 wherein the compression dictionary size selector is configured to select the supported compression dictionary size that is a minimum supported compression dictionary size responsive to available memory limitations.
13. The apparatus of claim 8 wherein the combiner is configured to select the ones of the updates enqueued for transmission such that aggregate size of the selected ones of the updates is less than or equal to, and as close as possible to, the selected compression dictionary size.
14. The apparatus of claim 8 wherein the combiner is configured to determine the aggregate size of the replica updates enqueued for transmission based on a number of populated data buffers and data buffer size.
15. A computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for generating compressed replication data, the method comprising:
repeatedly, and responsive to replica updates being enqueued for transmission to a mirror:
determining an aggregate size of the replica updates enqueued for transmission;
selecting a compression dictionary size based on the aggregate size of the replica updates enqueued for transmission;
selecting ones of the replica updates enqueued for transmission based on the selected compression dictionary size;
coalescing the selected ones of the replica updates;
compressing the coalesced replica updates using the selected compression dictionary size; and
providing the compressed coalesced replica updates to the mirror.
16. The computer-readable storage medium of claim 15 wherein the method comprises selecting the compression dictionary size from a plurality of supported compression dictionary sizes.
17. The computer-readable storage medium of claim 16 wherein the method comprises selecting the supported compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the replica updates enqueued for transmission.
18. The computer-readable storage medium of claim 16 wherein the method comprises selecting the supported compression dictionary size that is a maximum supported compression dictionary size responsive to one of the supported compression dictionary sizes being greater than or equal to the aggregate size of the replica updates enqueued for transmission.
19. The computer-readable storage medium of claim 16 wherein the method comprises selecting the supported compression dictionary size that is a minimum supported compression dictionary size.
20. The computer-readable storage medium of claim 15 wherein the method comprises selecting the ones of the replica updates enqueued for transmission such that aggregate size of the selected ones of the replica updates is less than or equal to, and as close as possible to, the selected compression dictionary size.