Chunked memory efficient storage data transfers
Chunked data transfers may include: receiving, from an initiator, a read command for first data stored in a storage system; allocating a first buffer of a predefined size; until all of the first data has been sent to the initiator: loading a portion of the first data from the storage system into the first buffer; and sending, to the initiator, the portion of the first data in the first buffer.
1. A method comprising:
receiving, from an initiator, a read command for first data stored in a storage system;
in response to receiving the read command, allocating, by the storage system, for sending one or more portions of the first data, at least a first data buffer of a plurality of data buffers that each have an amount of allocated computer memory that is smaller than a size of the first data; and
until all of the first data has been sent to the initiator:
loading a portion of the first data from the storage system into a first data buffer of the plurality of data buffers;
while the first data is being sent, loading one or more other portions of the first data from the storage system into one or more other data buffers of the plurality of data buffers; and
sending, to the initiator, the one or more other portions using the one or more other data buffers of the plurality of data buffers.
2. The method of claim 1 further comprising, until all of the first data has been sent to the initiator: sending, to the initiator, an other portion of the first data loaded from the storage system into an allocated second data buffer less than the size of the first data in the storage system.
3. The method of claim 2 , wherein loading the other portion of the first data into the second data buffer is performed at least partially concurrent to sending, to the initiator, the portion of the first data in the first data buffer, and wherein sending, to the initiator, the portion of the first data in the second data buffer performed at least partially concurrent to loading the portion of the first data into the first data buffer.
4. The method of claim 1 further comprising maintaining a data structure indicating a last sent portion of the first data.
5. The method of claim 4 , wherein loading the portion of the first data from the storage system into the first data buffer is based on the data structure.
6. The method of claim 1 further comprising:
receiving, from the initiator, a write command to store second data in the storage system;
allocating a second amount of computer memory as a second data buffer having a second memory size less than the size of the first data; and
until all of the second data has been stored in the storage system:
sending, to the initiator, a request for a portion of the second data;
storing the portion of the second data into the second data buffer; and
storing the portion of the second data from the second data buffer into the storage system.
7. An apparatus comprising a computer processor, a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
receiving, from an initiator, a read command for first data stored in a storage system;
in response to receiving the read command, allocating, by the storage system, for sending one or more portions of the first data, at least a first data buffer of a plurality of data buffers that each have an amount of allocated computer memory that is smaller than a size of the first data; and
until all of the first data has been sent to the initiator:
loading a portion of the first data from the storage system into a first data buffer of the plurality of data buffers;
while the first data is being sent, loading one or more other portions of the first data from the storage system into one or more other data buffers of the plurality of data buffers; and
sending, to the initiator, the one or more other portions using the one or more other data buffers of the plurality of data buffers.
8. The apparatus of claim 7 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of, until all of the first data has been sent to the initiator: sending, to the initiator, an other portion of the first data loaded from the storage system into an allocated second data buffer less than the size of the first data in the storage system.
9. The apparatus of claim 8 , wherein loading the other portion of the first data into the second data buffer is performed at least partially concurrent to sending, to the initiator, the portion of the first data in the first data buffer, and wherein sending, to the initiator, the portion of the first data in the second data buffer performed at least partially concurrent to loading the portion of the first data into the first data buffer.
10. The apparatus of claim 7 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the step of maintaining a data structure indicating a last sent portion of the first data.
11. The apparatus of claim 10 , wherein loading the portion of the first data from the storage system into the first data buffer is based on the data structure.
12. The apparatus of claim 7 further comprising computer program instructions that, when executed by the computer processor, cause the apparatus to carry out the steps of:
receiving, from the initiator, a write command to store second data in the storage system;
allocating a second amount of computer memory as a second data buffer having a second memory size less than the size of the first data; and
until all of the second data has been stored in the storage system:
sending, to the initiator, a request for a portion of the second data;
storing the portion of the second data into the second data buffer; and
storing the portion of the second data from the second data buffer into the storage system.
13. A computer program product disposed upon a computer readable medium, the computer program product comprising computer program instructions that, when executed, cause a computer to carry out the steps of:
receiving, from an initiator, a read command for first data stored in a storage system;
in response to receiving the read command, allocating, by the storage system, for sending one or more portions of the first data, a first data buffer that has an amount of allocated computer memory that is smaller than a size of the first data; and
until all of the first data has been sent to the initiator:
loading a portion of the first data from the storage system into a first data buffer of the plurality of data buffers;
while the first data is being sent, loading one or more other portions of the first data from the storage system into one or more other data buffers of the plurality of data buffers; and
sending, to the initiator, the one or more other portions using the one or more other data buffers of the plurality of data buffers.
14. The computer program product of claim 13 further comprising computer program instructions that, when executed, cause the computer to carry out the steps of, until all of the first data has been sent to the initiator: sending, to the initiator, an other portion of the first data loaded from the storage system into an allocated second data buffer less than the size of the first data in the storage system.
15. The computer program product of claim 14 , wherein loading the other portion of the first data into the second data buffer is performed at least partially concurrent to sending, to the initiator, the portion of the first data in the first data buffer, and wherein sending, to the initiator, the portion of the first data in the second data buffer performed at least partially concurrent to loading the portion of the first data into the first data buffer.
16. The computer program product of claim 13 further comprising computer program instructions that, when executed, cause the computer to carry out the step of maintaining a data structure indicating a last sent portion of the first data.
17. The computer program product of claim 16 , wherein loading the portion of the first data from the storage system into the first data buffer is based on the data structure.