Flexible remote data mirroring
Methods, systems, and configured storage media are provided for flexible data mirroring. In particular, the invention provides many-to-one data mirroring, including mirroring from local servers running the same or different operating systems and/or file systems at two or more geographically dispersed locations. The invention also provides one-to-many data mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a dedicated server or another server at the destination(s) to assist the remote mirroring unit(s). In addition, the invention provides flexibility by permitting the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data. Spoofing, SCSI and other bus emulations, and further tools and techniques are used in various embodiments of the invention.
1. A method for storing data from a host computing system, the method comprising:
receiving, at a storage system operationally independent from the host computing system, a plurality of blocks of the data, the received plurality of blocks of the data representing a change in the contents of a file from a first state of a storage volume to a second state of the storage volume, wherein the file comprises one or more blocks in each of the storage volume's first and second states, the received blocks not containing all of the information in any of the one or more blocks comprising the file in the second state of the storage volume, wherein each of the received plurality of blocks of the data is associated with a value indicating an order, and wherein each value associated with a selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block;
storing each of the received plurality of blocks of the data to a persistent data storage device;
receiving an indication of a particular location in the sequence; and
in response to receiving the indication of the particular location in the sequence:
retrieving, from the persistent data storage device, based on the associated values that correspond to the sequence, one or more distinguished blocks of the data of the stored plurality of blocks of the data, the one or more distinguished blocks of the data comprising less than all of the stored plurality of blocks of the data and selected based on the particular location in the sequence; and
providing the retrieved one or more distinguished blocks of the data.
2. The method of claim 1 wherein the associated values indicating an order are stored in a plurality of meta-data blocks, each meta-data block associated with one of the plurality of blocks of the data.
3. The method of claim 1 further comprising receiving and storing a snapshot of the storage volume in the first state of the storage volume.
4. The method of claim 3 ,
wherein the persistent data storage device is a first-in first-out queue;
wherein storing at least one of the plurality of blocks of the data to the first-in first-out queue comprises:
selecting a block of the plurality of blocks of the data that has already been stored to the first-in first-out queue,
wherein the selected block is selected based on the selected block being associated with a value that corresponds to an oldest position in the sequence, of the values that correspond to the plurality of blocks of the data that have already been stored to the first-in first-out queue;
updating the stored snapshot with the selected block;
removing the selected block from the first-in first-out queue; and
storing the at least one of the plurality of blocks of the data to the first-in first-out queue.
5. The method of claim 1 further comprising:
receiving and storing a snapshot of the storage volume in the first state of the storage volume; and
using the provided one or more distinguished blocks of the data to create a structure representing a state of the storage volume corresponding to the particular location in the sequence by applying the provided one or more distinguished blocks of the data to the snapshot of the storage volume.
6. The method of claim 1 further comprising:
sending one or more acknowledgement messages, wherein each acknowledgement message indicates that one or more of the plurality of blocks of the data has been stored to the persistent data storage device.
7. The method of claim 6 wherein each acknowledgement message includes a signature corresponding to the block of the data that has been acknowledged as stored to the persistent data storage device.
8. The method of claim 1 wherein the value indicating an order is a timestamp.
9. The method of claim 1 ,
wherein the persistent data storage device implements a first-in first-out queue;
wherein storing each of the plurality of blocks of the data to the persistent data storage device comprises storing each of the plurality of blocks of the data to the first-in first-out queue; and
wherein the method further comprises moving at least some of the of the plurality of blocks of the data to another portion of the persistent data storage device from the first-in first-out queue in the same order the plurality of blocks of the data were stored to the first-in first-out queue.
10. The method of claim 1 , wherein retrieving one or more distinguished blocks of data comprises using binary search according to the associated values for each data block, wherein each value indicates an order, and wherein each value associated with the selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block.
11. A computer-readable storage medium that is not a transitory signal, storing instructions that, when executed by a computing system, cause the computing system to perform operations for storing data from a host computing system, the operations comprising:
receiving, at a storage system operationally independent from the host computing system, a plurality of blocks of the data, the received plurality of blocks of the data representing a change in the contents of a file from a first state of a storage volume to a second state of the storage volume, wherein the file comprises one or more blocks in each of the storage volume's first and second states, the received blocks not containing all of the information in any of the one or more blocks comprising the file in the second state of the storage volume, wherein each of the received plurality of blocks of the data is associated with a value indicating an order, and wherein each value associated with a selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block;
storing each of the received plurality of blocks of the data to a persistent data storage device;
receiving an indication of a particular location in the sequence; and
in response to receiving the indication of the particular location in the sequence:
retrieving, from the persistent data storage device, based on the associated values that correspond to the sequence, one or more distinguished blocks of the data of the stored plurality of blocks of the data, the one or more distinguished blocks of the data comprising less than all of the stored plurality of blocks of the data and selected based on the particular location in the sequence; and
providing the retrieved one or more distinguished blocks of the data.
12. The computer-readable storage medium of claim 11 wherein the associated values indicating an order are stored in a plurality of meta-data blocks, each meta-data block associated with one of the plurality of blocks of the data.
13. The computer-readable storage medium of claim 11 wherein the operations further comprise receiving and storing a snapshot of the storage volume in the first state of the storage volume.
14. The computer-readable storage medium of claim 13 ,
wherein the persistent data storage device implements a first-in first-out queue;
wherein storing each of the plurality of blocks of the data to the persistent data storage device comprises storing each of the plurality of blocks of the data to the first-in first-out queue;
wherein storing at least one of the plurality of blocks of the data to the first-in first-out queue comprises:
selecting a block of the plurality of blocks of the data that has already been stored to the first-in first-out queue,
wherein the selected block is selected based on the selected block being associated with a value that corresponds to an oldest position in the sequence, of the values that correspond to the plurality of blocks of the data that have already been stored to the first-in first-out queue;
updating the stored snapshot with the selected block;
removing the selected block from the first-in first-out queue; and
storing the at least one of the plurality of blocks of the data to the first-in first-out queue.
15. The computer-readable storage medium of claim 11 wherein the operations further comprise:
receiving and storing a snapshot of the storage volume in the first state of the storage volume; and
using the provided one or more distinguished blocks of the data to create a structure representing a state of the storage volume corresponding to the particular location in the sequence by applying the provided one or more distinguished blocks of the data to the snapshot of the storage volume.
16. The computer-readable storage medium of claim 11 wherein the operations further comprise:
sending one or more acknowledgement messages, wherein each acknowledgement message indicates that one or more of the plurality of blocks of the data has been stored to the persistent data storage device.
17. The computer-readable storage medium of claim 11 wherein the operations further comprise:
performing binary search to locate one or more distinguished blocks of data, wherein each data block is associated with a value, wherein each value indicates an order, and wherein each value associated with the selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block.
18. A storage system that is operationally independent from a host computing system, the storage system comprising:
a persistent data storage device;
one or more processors; and
a network interface;
wherein the storage system is configured to:
receive, through the network interface, a plurality of blocks of the data, the received plurality of blocks of the data representing a change in the contents of a file from a first state of a storage volume to a second state of the storage volume, wherein the file comprises one or more blocks in each of the storage volume's first and second states, the received blocks not containing all of the information in any of the blocks comprising the file in the second state of the storage volume, wherein each of the received plurality of blocks of the data is associated with a value indicating an order, and wherein each value associated with a selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block;
store each of the received plurality of blocks of the data to the persistent data storage device;
receive, through the network interface, an indication of a particular location in the sequence; and
in response to receiving the indication of the particular location in the sequence:
retrieve, from the persistent data storage device, based on the associated values that correspond to the sequence, one or more distinguished blocks of the data of the stored plurality of blocks of the data, the one or more distinguished blocks of the data comprising less than all of the stored plurality of blocks of the data and selected based on the particular location in the sequence; and
provide the retrieved one or more distinguished blocks of the data.
19. The storage system of claim 18 , wherein the associated values indicating an order are stored in a plurality of meta-data blocks, each meta-data block associated with one of the plurality of blocks of the data.
20. The storage system of claim 18 , wherein the storage system is further configured to receive and store a snapshot of the storage volume in the first state of the storage volume.
21. The storage system of claim 20 ,
wherein the persistent data storage device is a first-in first-out queue;
wherein storing at least one of the plurality of blocks of the data to the first-in first-out queue comprises:
selecting a block of the plurality of blocks of the data that has already been stored to the first-in first-out queue,
wherein the selected block is selected based on the selected block being associated with a value that corresponds to an oldest position in the sequence, of the values that correspond to the plurality of blocks of the data that have already been stored to the first-in first-out queue;
updating the stored snapshot with the selected block;
removing the selected block from the first-in first-out queue; and
storing the at least one of the plurality of blocks of the data to the first-in first-out queue.
22. The storage system of claim 18 , wherein the storage system is further configured to:
receive and store a snapshot of the storage volume in the first state of the storage volume; and
use the provided one or more distinguished blocks of the data to create a structure representing a state of the storage volume corresponding to the particular location in the sequence by applying the provided one or more distinguished blocks of the data to the snapshot of the storage volume.
23. The storage system of claim 19 , wherein the storage system is further configured to:
perform binary search to locate one or more distinguished blocks of data, utilizing the meta-data blocks associated with one or more of the plurality of the blocks of data, wherein each meta-data block stores a value indicating an order, and wherein each value associated with the selected data block corresponds to a position in a sequence in which the host computing system stores contents corresponding to the selected data block.