System and method for data transfer using a recoverable data pipe
View Patent ↗A system for data transfer using a recoverable data pipe includes a data producer, one or more data consumers, a storage device and a data pipe manager. The data producer may be configured to append updated data blocks to the storage device via a producer storage input/output (I/O) channel. The data consumers may be configured to read data blocks from the storage device via consumer storage I/O channels. The data pipe manager may be configured to maintain metadata identifying data blocks of the storage device that have been read by each data consumer, and to release backing storage corresponding to the data blocks that have been read by all data consumers.
1. A system comprising:
a data producer;
one or more data consumers;
a storage device; and
a data pipe manager;
wherein the data producer is configured to:
append updated data blocks to the storage device via a producer storage input/output (I/O) channel;
wherein each data consumer of the one or more data consumers is configured to:
read data blocks of the storage device via a consumer storage I/O channel;
and wherein the data pipe manager is configured to:
maintain metadata identifying data blocks of the storage device that have been read by each data consumer of the one or more data consumers; and
release backing storage corresponding to the data blocks of the storage device that have been read by each data consumer of the one or more data consumers.
2. The system as recited in claim 1 , wherein a data consumer of the one or more data consumers is configured to read a first data block of the storage device concurrently with the data producer appending a second data block to the storage device.
3. The system as recited in claim 1 , wherein the data pipe manager is configured to release the backing storage concurrently with the data producer appending a data block to the storage device.
4. The system as recited in claim 1 , wherein the data pipe manager is further configured to:
allocate released backing storage to store additional updated data blocks appended by the data producer.
5. The system as recited in claim 1 , wherein the storage device comprises a logical volume.
6. The system as recited in claim 5 , wherein the logical volume is striped across a plurality of physical storage devices.
7. The system as recited in claim 1 , wherein the metadata comprises an indicator identifying a latest updated data block of the storage device.
8. The system as recited in claim 7 , wherein the data pipe manager is further configured to:
perform a first physical write operation including a metadata update corresponding to an appended data block concurrently with a second physical write operation including contents of the appended data block.
9. The system as recited in claim 1 , wherein the metadata comprises an indicator identifying a region of the storage device that is currently backed by persistent storage.
10. The system as recited in claim 1 , wherein the data pipe manager is configured to maintain the metadata within the storage device.
11. The system as recited in claim 1 , wherein the data producer is configured to append a plurality of updated data blocks to the storage device as part of a single atomic transaction.
12. A method comprising:
a data producer appending updated data blocks to a storage device via a producer storage input/output (I/O) channel;
each data consumer of one or more data consumers reading data blocks of the storage device via a respective consumer storage I/O channel;
maintaining metadata identifying data blocks of the storage device that have been read by each data consumer of the one or more data consumers; and
releasing backing storage corresponding to data blocks of the storage device that have been read by each data consumer of the one or more data consumers.
13. The method as recited in claim 12 , wherein a data consumer of the one or more data consumers is configured to read a first data block of the storage device concurrently with the data producer appending a second data block to the storage device.
14. The method as recited in claim 12 , wherein the backing storage is released concurrently with the data producer appending a data block to the storage device.
15. The method as recited in claim 12 , further comprising:
allocating released backing storage to store additional updated data blocks appended by the data producer.
16. The method as recited in claim 12 , wherein the storage device comprises a logical volume.
17. The method as recited in claim 12 , wherein the metadata comprises an indicator identifying a latest updated data block of the storage device, further comprising:
performing a first physical write operation including a metadata update corresponding to an appended data block concurrently with a second physical write operation including contents of the appended data block.
18. The method as recited in claim 12 , further comprising:
maintaining the metadata within the storage device.
19. A computer accessible storage medium comprising program instructions, wherein the instructions are executable to:
append updated data blocks to a storage device via a producer storage input/output (I/O) channel from a data producer;
read data blocks of the storage device via a respective consumer storage I/O channel at each data consumer of one or more data consumers;
maintain metadata identifying data blocks of the storage device that have been read by each data consumer of the one or more data consumers; and
release backing storage corresponding to data blocks of the storage device that have been read by each data consumer of the one or more data consumers.
20. The computer accessible medium as recited in claim 19 , wherein the instructions are further executable to:
read a first data block of the storage device at a data consumer of one or more data consumers concurrently with the appending of a second data block to the storage device from the data producer.
21. The computer accessible medium as recited in claim 19 , wherein the instructions are further executable to:
release the backing storage is released concurrently with the appending of a data block to the storage device from a data producer.
22. The computer accessible medium as recited in claim 19 , wherein the instructions are further executable to:
allocate released backing storage to store additional updated data blocks appended from the data producer.
23. The computer accessible medium as recited in claim 19 , wherein the metadata comprises an indicator identifying a latest updated data block of the storage device, wherein the instructions are further executable to:
perform a first physical write operation including a metadata update corresponding to an appended data block concurrently with a second physical write operation including contents of the appended data block.
24. The computer accessible medium as recited in claim 19 , further comprising:
maintaining the metadata within the storage device.