Eventually consistent synchronous data replication in a storage system
Described embodiments provide systems and methods for operating a storage system. One or more production volumes of the storage system are selected for continuous replication. A number, N, is selected that is associated with a number of damaged volumes the storage system can sustain and maintain data consistency. Write transactions from a host to an associated one of the selected one or more production volumes are intercepted. The intercepted write transactions are sent to the associated production volume and to a plurality of copy volumes. When acknowledgments of the write transaction have been received from N copy volumes, the write transaction is acknowledged to the host.
1. A method comprising:
selecting one or more production volumes of a storage system for continuous replication;
selecting a number, N, associated with a number of damaged volumes the storage system can sustain and maintain data consistency;
intercepting one or more write transactions from a host to an associated one of the selected one or more production volumes;
sending the intercepted one or more write transactions to the associated production volume and to a plurality of copy volumes;
when acknowledgments of the one or more write transactions have been received from N copy volumes, acknowledging the one or more write transactions to the host;
assigning each of the intercepted one or more write transactions a unique transaction number, the transaction number monotonically increasing based upon an order in which the intercepted one or more write transactions are intercepted; and
verifying one or more of the plurality of copy volumes as synchronized with an associated production volume based upon the transaction numbers of the intercepted one or more write transactions written to each of the copy volumes.
2. The method of claim 1 , wherein verifying one or more copy volumes as synchronized with an associated production volume comprises:
determining a highest transaction number written to each copy volume;
determining one or more copy volumes having a highest transaction number; and
indicating the determined one or more copy volumes as being synchronized with the associated production volume.
3. The method of claim 1 , wherein an acknowledgment of a write transaction having an associated transaction number implies acknowledgment of write transactions having transactions numbers lower than the associated transaction number.
4. The method of claim 1 , further comprising:
determining one or more write transactions received by all the copy volumes;
tracking metadata associated with write transactions received by fewer than all the copy volumes; and
generating at least one synchronous copy volume based upon the tracked metadata and the one or more write transactions received by all the copy volumes.
5. The method of claim 4 , wherein the metadata is tracked in one or more DO METADATA journals, each journal associated with a given copy volume, and wherein generating at least one synchronous copy volume based upon the tracked metadata and the one or more write transactions received by all the copy volumes comprises merging the DO METADATA of each copy volume with the write transactions that were received by all the copy volumes.
6. The method of claim 1 , wherein N is an integer greater than zero and less than or equal to a sum of a number of production volumes and a number of copy volumes of the storage system.
7. The method of claim 1 , further comprising receiving acknowledgments to the write transaction from one or more additional copy volumes as a background process while processing new write transactions.
8. A system comprising:
a processor; and
memory storing computer program code that when executed on the processor causes the processor to operate a storage system, the storage system operable to perform the operations of:
selecting one or more production volumes of a storage system for continuous replication;
selecting a number, N, associated with a number of damaged volumes the storage system can sustain and maintain data consistency, wherein N is an integer greater than zero and less than or equal to a sum of a number of production volumes and a number of copy volumes of the storage system;
intercepting one or more write transactions from a host to an associated one of the selected one or more production volumes;
sending the intercepted one or more write transactions to the associated production volume and to the copy volumes; and
when acknowledgments of the one or more write transactions have been received from N copy volumes, acknowledging the one or more write transactions to the host;
assigning each of the intercepted one or more write transactions a unique transaction number, the transaction number monotonically increasing based upon an order in which the one or more write transactions are intercepted; and
verifying one or more of the plurality of copy volumes as synchronized with an associated production volume based upon the transaction numbers of write transactions written to each of the copy volumes.
9. The system of claim 8 , wherein the storage system is further operable to perform the operations of:
determining a highest transaction number written to each copy volume;
determining one or more copy volumes having a highest transaction number; and
indicating the determined one or more copy volumes as being synchronized with the associated production volume.
10. The system of claim 8 , wherein an acknowledgment of a write transaction having an associated transaction number implies acknowledgment of write transactions having transactions numbers lower than the associated transaction number.
11. The system of claim 8 , wherein the storage system is further operable to perform the operations of:
determining one or more write transactions received by all the copy volumes;
tracking metadata associated with write transactions received by fewer than all the copy volumes; and
generating at least one synchronous copy volume based upon the tracked metadata and the one or more write transactions received by all the copy volumes.
12. The system of claim 11 , wherein the metadata is tracked in one or more DO METADATA journals, each journal associated with a given copy volume, and wherein the storage system is further operable to perform the operation of merging the DO METADATA of each copy volume with the write transactions that were received by all the copy volumes.
13. The system of claim 8 , wherein the storage system is further operable to perform the operation of receiving acknowledgments to the write transaction from one or more additional copy volumes as a background process while processing new write transactions.
14. A computer program product including a non-transitory computer readable storage medium having computer program code encoded thereon that when executed on a processor of a computer causes the computer to operate a storage system, the computer program product comprising:
computer program code for selecting one or more production volumes of a storage system for continuous replication;
computer program code for selecting a number, N, associated with a number of damaged volumes the storage system can sustain and maintain data consistency, wherein N is an integer greater than zero and less than or equal to a sum of a number of production volumes and a number of copy volumes of the storage system;
computer program code for intercepting one or more write transactions from a host to an associated one of the selected one or more production volumes;
computer program code for sending the intercepted one or more write transactions to the associated production volume and to the copy volumes;
when acknowledgments of the one or more write transactions have been received from N copy volumes, computer program code for acknowledging the one or more write transactions to the host;
computer program code for assigning each of the intercepted one or more write transactions a unique transaction number, the transaction number monotonically increasing based upon an order in which the one or more write transactions are intercepted; and
computer program code for verifying one or more of the plurality of copy volumes as synchronized with an associated production volume based upon the transaction numbers of the one or more write transactions written to each of the copy volumes.
15. The computer program product of claim 14 , further comprising:
computer program code for determining a highest transaction number written to each copy volume;
computer program code for determining one or more copy volumes having a highest transaction number; and
computer program code for indicating the determined one or more copy volumes as being synchronized with the associated production volume.
16. The computer program product of claim 14 , wherein an acknowledgment of a write transaction having an associated transaction number implies acknowledgment of write transactions having transactions numbers lower than the associated transaction number.
17. The computer program product of claim 14 , further comprising:
computer program code for determining one or more write transactions received by all the copy volumes;
computer program code for tracking metadata associated with write transactions received by fewer than all the copy volumes; and
computer program code for generating at least one synchronous copy volume based upon the tracked metadata and the one or more write transactions received by all the copy volumes.