Host data replication allocating single memory buffers to store multiple buffers of received host data and to internally process the received host data
On a local data storage system, a single buffer of contiguous local memory is allocated to store host data from multiple received host data buffers in a local cache, and local internal data processing operations may be performed on the host data with local data processing resources using the single buffer of contiguous local memory in the local memory to store the host data. On a remote data storage system onto which the host data is replicated, a single buffer of contiguous remote memory is allocated to store host data from multiple transmission units used to convey the host data to the remote data storage system, and remote internal data processing operations may be performed on the host data by one or more remote data processing resources using the single buffer of contiguous remote memory to store the host data from the received transmission units.
1. A method of performing host data replication between a local data storage system and a remote data storage system, comprising:
storing host data received by the local data storage system from one or more host computers into a plurality of discontiguous received host data buffers within a local cache in the local data storage system, wherein contents of the host data buffers within the local cache is also stored into non-volatile storage of the local data storage system to persistently store the contents of the host data buffers in the local data storage system;
generating a local data buffer location list having a plurality of entries, wherein each entry in the local data buffer location list indicates a location and a length of a corresponding one of the received host data buffers in the local cache; and
in response to the local data buffer location list:
allocating a single buffer of contiguous local memory in a memory of the local data storage system, wherein the single buffer of contiguous local memory has a size that is at least as large as a total size of the received host buffers, and
performing at least one local internal data processing operation on the host data with at least one local data processing resource within the local data storage system while using the single buffer of contiguous local memory to store the host data from the received host data buffers.
2. The method of claim 1 , wherein the at least one local data processing resource includes local direct memory access circuitry coupled to the local cache and the memory of the local data storage system, and wherein the at least one local internal data processing operation includes moving the host data from the host data buffers in the local cache to the single buffer of contiguous local memory using the local direct memory access circuitry.
3. The method of claim 2 , wherein moving the host data from the host data buffers in the local cache to the single buffer of contiguous local memory includes i) indicating, responsive to the entries in the local data buffer location list, the locations of the received host data buffers in the local cache to the local direct memory access circuitry, ii) indicating a location of the single buffer of contiguous local memory in the memory of the local data storage system to the local direct memory access circuitry, and iii) triggering the local direct memory access circuitry to begin moving the host data from the received host data buffers in the local cache to the single buffer of contiguous local memory.
4. The method of claim 3 , wherein the at least one local data processing resource includes local data compression circuitry coupled to the memory of the local data storage system, and wherein the at least one local internal data processing operation includes compressing the host data in the single buffer of contiguous local memory using the local data compression circuitry.
5. The method of claim 4 , wherein compressing the host data in the single buffer of contiguous local memory includes i) indicating the location of the single buffer of contiguous local memory in the memory of the local data storage system to the local data compression circuitry coupled to the memory of the local data storage system, and ii) triggering the local data compression circuitry to begin compressing the host data in the single buffer of contiguous local memory.
6. The method of claim 5 , wherein the at least one local data processing resource includes at least one local central processing unit, and wherein the at least one local internal data processing operation includes executing a network controller module on the local central processing unit to add network header information to the host data in the single buffer of contiguous memory in the memory of the local data storage system, wherein the network header information includes mapping information indicating locations within a remote cache of the remote data storage system into which the host data is to be stored.
7. The method of claim 6 , wherein the at least one local data processing resource further includes a local network interface controller, and wherein the at least one local internal data processing operation includes executing a TCP/IP module on the local central processing unit to cause a large send offload operation to be performed by a local network interface card in the local data storage system by passing the single buffer of contiguous memory to the local network interface controller to cause the local network interface controller to i) divide the host data stored in the single buffer of contiguous memory into packets, and ii) transmit the packets as transmission units to the remote data storage system.
8. The method of claim 6 , further comprising:
receiving the transmission units by the remote data storage system;
allocating a single buffer of contiguous remote memory in a memory of the remote data storage system, wherein the single buffer of contiguous remote memory has a size that is at least as large as a total size of the received transmission units; and
performing at least one remote internal data processing operation on the host data with at least one remote data processing resource within the remote data storage system while using the single buffer of contiguous remote memory to store the host data contained within the received transmission units.
9. The method of claim 8 , wherein the at least one remote data processing resource includes remote data compression circuitry coupled to the memory of the remote data storage system, and wherein the at least one remote internal data processing operation includes decompressing the host data in the single buffer of contiguous remote memory using the remote data compression circuitry.
10. The method of claim 9 , wherein the at least one remote data processing resource includes remote direct memory access circuitry coupled to the remote cache and the memory of the remote data storage system, and further comprising:
generating, at the remote data storage system from mapping information in transmission unit metadata, a remote data buffer location list having a plurality of entries, wherein each entry in the remote data buffer location list indicates a location and a length of a buffer within the remote cache of the remote data storage system into which the replicated host data is to be stored, and wherein the replicated host data in the remote cache is also stored into non-volatile storage of the remote data storage system to persistently store the replicated host data on the remote data storage system; and
wherein the at least one remote internal data processing operation includes moving the host data from the single buffer of contiguous remote memory to the locations within the remote cache of the remote data storage system into which the replicated host data is to be stored, responsive to the entries in the remote data buffer location list, using the remote direct memory access circuitry.
11. A system for replicating host data, comprising:
a local data storage system having processing circuitry and at least one memory with program code stored thereon for execution on the processing circuitry, wherein the local data storage system is configured and arranged to:
store host data received by the local data storage system from one or more host computers into a plurality of discontiguous received host data buffers within a local cache in the local data storage system, wherein contents of the host data buffers within the local cache is also stored into non-volatile storage of the local data storage system to persistently store the contents of the host data buffers in the local data storage system;
generate a local data buffer location list having a plurality of entries, wherein each entry in the local data buffer location list indicates a location and a length of a corresponding one of the received host data buffers in the local cache; and
in response to the local data buffer location list:
allocate a single buffer of contiguous local memory in a memory of the local data storage system, wherein the single buffer of contiguous local memory has a size that is at least as large as a total size of the received host buffers, and
perform at least one local internal data processing operation on the host data with at least one local data processing resource within the local data storage system while using the single buffer of contiguous local memory to store the host data from the received host data buffers.
12. The system of claim 11 , wherein the at least one local data processing resource includes local direct memory access circuitry coupled to the local cache and the memory of the local data storage system, and wherein the local data storage system is further configured and arranged to perform the at least one local internal data processing operation at least in part by moving the host data from the host data buffers in the local cache to the single buffer of contiguous local memory using the local direct memory access circuitry.
13. The system of claim 12 , wherein the local data storage system is further configured and arranged to move the host data from the host data buffers in the local cache to the single buffer of contiguous local memory at least in part by i) indicating, responsive to the entries in the local data buffer location list, the locations of the received host data buffers in the local cache to the local direct memory access circuitry, ii) indicating a location of the single buffer of contiguous local memory in the memory of the local data storage system to the local direct memory access circuitry, and iii) triggering the local direct memory access circuitry to begin moving the host data from the received host data buffers in the local cache to the single buffer of contiguous local memory.
14. The system of claim 13 , wherein the at least one local data processing resource includes local data compression circuitry coupled to the memory of the local data storage system, and wherein the local data storage system is further configured and arranged to perform the at least one local internal data processing operation at least in part by compressing the host data in the single buffer of contiguous local memory using the local data compression circuitry.
15. The system of claim 14 , wherein the local data storage system is further configured and arranged to compress the host data in the single buffer of contiguous local memory at least in part by i) indicating the location of the single buffer of contiguous local memory in the memory of the local data storage system to the local data compression circuitry coupled to the memory of the local data storage system, and ii) triggering the local data compression circuitry to begin compressing the host data in the single buffer of contiguous local memory.
16. The system of claim 15 , further comprising:
a remote data storage system having processing circuitry and at least one memory with program code stored thereon for execution on the processing circuitry; and
wherein the at least one local data processing resource includes at least one local central processing unit, and wherein the local data storage system is further configured and arranged to perform the at least one local internal data processing operation at least in part by executing a network controller module on the local central processing unit to add network header information to the host data in the single buffer of contiguous memory in the memory of the local data storage system, wherein the network header information includes mapping information indicating locations within a remote cache of the remote data storage system into which the host data is to be stored.
17. The system of claim 16 , wherein the at least one local data processing resource further includes a local network interface controller, and wherein the local data storage system is further configured and arranged to perform the at least one local internal data processing operation at least in part by executing a TCP/IP module on the local central processing unit to cause a large send offload operation to be performed by a local network interface card in the local data storage system by passing the single buffer of contiguous memory to the local network interface controller to cause the local network interface controller to i) divide the host data stored in the single buffer of contiguous memory into packets, and ii) transmit the packets as transmission units to the remote data storage system.
18. The system of claim 16 , wherein the remote data storage system is configured and arranged to:
receive the transmission units;
allocate a single buffer of contiguous remote memory in the memory of the remote data storage system, wherein the single buffer of contiguous remote memory has a size that is at least as large as a total size of the received transmission units; and
perform at least one remote internal data processing operation on the host data with at least one remote data processing resource within the remote data storage system while using the single buffer of contiguous remote memory to store the host data contained within the received transmission units.
19. The system of claim 18 , wherein the at least one remote data processing resource includes remote data compression circuitry coupled to the memory of the remote data storage system, and wherein the remote data storage system is further configured and arranged to perform the at least one remote internal data processing operation at least in part by decompressing the host data in the single buffer of contiguous remote memory using the remote data compression circuitry.
20. The system of claim 19 , wherein the at least one remote data processing resource includes remote direct memory access circuitry coupled to the remote cache and the memory of the remote data storage system, and wherein the remote data storage system is further configured and arranged to:
generate, from mapping information in transmission unit metadata, a remote data buffer location list having a plurality of entries, wherein each entry in the remote data buffer location list indicates a location and a length of a buffer within the remote cache of the remote data storage system into which the replicated host data is to be stored, and wherein the replicated host data in the remote cache is also stored into non-volatile storage of the remote data storage system to persistently store the replicated host data on the remote data storage system; and
perform the at least one remote internal data processing operation at least in part by moving the host data from the single buffer of contiguous remote memory to the locations within the remote cache of the remote data storage system into which the replicated host data is to be stored, responsive to the entries in the remote data buffer location list, using the remote direct memory access circuitry.
21. A computer program product, comprising:
at least one non-transitory computer readable medium storing program code for, the program code, when executed by processing circuitry, causing the processing circuitry to perform the steps of:
storing host data received by the local data storage system from one or more host computers into a plurality of discontiguous received host data buffers within a local cache in the local data storage system, wherein contents of the host data buffers within the local cache is also stored into non-volatile storage of the local data storage system to persistently store the contents of the host data buffers in the local data storage system;
generating a local data buffer location list having a plurality of entries, wherein each entry in the local data buffer location list indicates a location and a length of a corresponding one of the received host data buffers in the local cache; and
in response to the local data buffer location list:
allocating a single buffer of contiguous local memory in a memory of the local data storage system, wherein the single buffer of contiguous local memory has a size that is at least as large as a total size of the received host buffers, and
performing at least one local internal data processing operation on the host data with at least one local data processing resource within the local data storage system while using the single buffer of contiguous local memory to store the host data from the received host data buffers.