Data storage system with core-affined thread processing of data movement requests

1. A method of operating a data storage system to process data movement requests, the data storage system having processing circuitry including one or more multi-core processors each having a respective plurality of processing cores, comprising:

for each data movement request, executing a multi-threaded library component and underlying device drivers to perform a data movement operation including reading data from a source storage device into a source area of memory, copying data from the source area to a destination area of memory, and writing data from the destination area to a destination storage device, the library component having an initial operation of invoking an underlying driver to initiate the reading of data, and having subsequent operations of (1) in response to completion of data reading by an underlying driver, performing the copying of data, (2) invoking an underlying driver to initiate the writing of data, and (3) in response to completion of data writing by an underlying driver, initiating additional data movement operations to complete data movement for an entire range of the data movement request,

wherein executing the multi-threaded component includes:

creating per-core queues for holding the data movement requests during processing, each queue uniquely associated with a respective core of the processing circuitry for holding data movement requests being processed by the associated core;

creating per-core threads each being affined to a respective core and being a unit of executable code operable to execute library component operations for data movement requests of the queue associated with the affined core;

distributing the data movement requests among the per-core queues for parallel processing of the data movement requests by the respective core-affined threads; and

for each data movement request, executing the respective core-affined thread on only the affined core, including (1) initially starting the thread on the affined core to perform the initial operation, and (2) subsequently re-starting the thread on the affined core to perform each of the subsequent operations.

2. The method of claim 1 , wherein each core-affined thread has a multi-loop structure having three loops, a first loop iterating over aligned sub-ranges of a complete range of the data movement request, a second loop iterating over multiple read callbacks by which all data of a given sub-range is copied from the source storage device into memory, and a third loop iterating over multiple write callbacks by which all data for a given read callback is copied from memory to the destination storage device.

3. The method of claim 2 , wherein each iteration of the first loop includes sending a read I/O request packet (IRP) to an underlying driver and receiving a corresponding IRP completion from the underlying driver after the second and third loops have been executed, the sending of the read IRP initiating action of the underlying driver that in turn initiates execution of the second and third loops.

4. The method of claim 3 , wherein each iteration of the second loop includes (1) receiving a corresponding read callback from the underlying driver identifying a respective source scatter-gather list (SGL) storing data from the source storage device, the read callbacks occurring in response to the read IRP, (2) sending a write IRP to an underlying driver to initiate a writing operation to the destination storage, the sending of the write IRP initiating action of the underlying driver that in turn initiates execution of the third loop, (3) upon completion of the third loop, confirming the source SGL to the underlying driver, resulting in the underlying driver subsequently sending a write IRP completion that terminates the iteration of the second loop.

5. The method of claim 4 , wherein each iteration of the third loop includes (1) receiving a corresponding write callback from the underlying driver identifying a respective destination SGL to store data to be written to the destination storage device, the write callbacks occurring in response to the write IRP, (2) copying data from the source SGL to the destination SGL, and (3) confirming the destination SGL to the underlying driver.

6. The method of claim 1 , wherein each per-core thread is one of a set of threads collectively realizing implementing operations of the library component for a DM request, the set of threads including a copy thread, a callback thread, and an I/O request packet (IRP) completion thread, the copy thread being core-affined and being a main execution thread for each DM request, the IRP completion thread also being core-affined and performing IRP completion processing in response to completion of a read IRP and a write IRP initiated by the copy thread, the callback thread performing callback processing in response to a read callback and a write callback from the underlying drivers identifying the source area and destination area of memory for the data copying, both the callback thread and the IRP completion thread generating respective signals to awaken the copy thread to resume operation on a DM request after processing of a callback and IRP completion respectively are completed.

7. The method of claim 6 , wherein the callback thread is not core-affined and executes on a core other than the core on which the copy thread and IRP completion thread execute, and operation of the callback thread for a given data movement request includes (1) identifying which core the data movement request is assigned to, and (2) generating the signal to the copy thread affined to the core to which the data movement request is assigned.

8. The method of claim 6 , wherein the callback processing of the callback thread includes:

storing a source scatter-gather list (SGL) in association with the data movement request;

for a write callback, sending a signal to the copy thread to initiate data copying from the source area to the destination area; and

for a read callback, prepare a write IRP and send it to the underlying driver.

9. The method of claim 8 , wherein the copy thread responds to the signal by:

determining whether both source and destination SGLs are available; and

in response to both source destination SGLs being available:

performing a memory copy from source SGL to destination SGL;

confirming the destination SGL to the underlying driver;

determining whether all of the source SGL has been copied to a destination SGL, and if not then looping back to beginning to wait for more destination SGLs to be obtained; and

if all of the source SGL has been copied to a destination SGL, then confirming the source SGL to the underlying driver.

10. The method of claim 6 , wherein the IRP completion processing of the IRP completion thread includes:

for a write IRP completion:

(1) determining whether all the data in a source scatter-gather list (SGL) has been copied, and if not then preparing another write IRP and sending it to the underlying driver;

(2) if all the data in a source scatter-gather list (SGL) has been copied, then sending the signal the copy thread to await a read IRP completion; and

for a read IRP completion:

(1) determining whether all the data of the current data movement request has been copied, and if not then preparing another read IRP and sending it to the underlying driver; and

(2) if all the data of the current data movement request has been copied, then sending the signal the copy thread to complete the processing of the data movement request.

11. The method of claim 10 , wherein the copy thread responds to the signal by:

determining whether both the read IRP and write IRP are complete for a sub-range of the data movement request, and if not then awaiting completion of both the read IRP and write IRP;

if both the read IRP and write IRP are complete for the sub-range of the data movement request, then determining whether the processing for the entire data movement request has been performed, and if not then initiating additional device read and copy operations to complete the data movement request.

12. A data storage system having data storage devices and processing circuitry, the processing circuitry including one or more multi-core processors each having a respective plurality of processing cores, the processing circuitry executing computer program instructions to cause the data storage system to process data movement requests by:

for each data movement request, executing a multi-threaded library component and underlying device drivers to perform a data movement operation including reading data from a source storage device into a source area of memory, copying data from the source area to a destination area of memory, and writing data from the destination area to a destination storage device, the library component having an initial operation of invoking an underlying driver to initiate the reading of data, and having subsequent operations of (1) in response to completion of data reading by an underlying driver, performing the copying of data, (2) invoking an underlying driver to initiate the writing of data, and (3) in response to completion of data writing by an underlying driver, initiating additional data movement operations to complete data movement for an entire range of the data movement request,

wherein executing the multi-threaded component includes:

creating per-core queues for holding the data movement requests during processing, each queue uniquely associated with a respective core of the processing circuitry for holding data movement requests being processed by the associated core;

creating per-core threads each being affined to a respective core and being a unit of executable code operable to execute library component operations for data movement requests of the queue associated with the affined core;

distributing the data movement requests among the per-core queues for parallel processing of the data movement requests by the respective core-affined threads; and

for each data movement request, executing the respective core-affined thread on only the affined core, including (1) initially starting the thread on the affined core to perform the initial operation, and (2) subsequently re-starting the thread on the affined core to perform each of the subsequent operations.

13. The data storage system of claim 12 , wherein each core-affined thread has a multi-loop structure having three loops, a first loop iterating over aligned sub-ranges of a complete range of the data movement request, a second loop iterating over multiple read callbacks by which all data of a given sub-range is copied from the source storage device into memory, and a third loop iterating over multiple write callbacks by which all data for a given read callback is copied from memory to the destination storage device.

14. The data storage system of claim 13 , wherein each iteration of the first loop includes sending a read I/O request packet (IRP) to an underlying driver and receiving a corresponding IRP completion from the underlying driver after the second and third loops have been executed, the sending of the read IRP initiating action of the underlying driver that in turn initiates execution of the second and third loops.

15. The data storage system of claim 14 , wherein each iteration of the second loop includes (1) receiving a corresponding read callback from the underlying driver identifying a respective source scatter-gather list (SGL) storing data from the source storage device, the read callbacks occurring in response to the read IRP, (2) sending a write IRP to an underlying driver to initiate a writing operation to the destination storage, the sending of the write IRP initiating action of the underlying driver that in turn initiates execution of the third loop, (3) upon completion of the third loop, confirming the source SGL to the underlying driver, resulting in the underlying driver subsequently sending a write IRP completion that terminates the iteration of the second loop.

16. The data storage system of claim 15 , wherein each iteration of the third loop includes (1) receiving a corresponding write callback from the underlying driver identifying a respective destination SGL to store data to be written to the destination storage device, the write callbacks occurring in response to the write IRP, (2) copying data from the source SGL to the destination SGL, and (3) confirming the destination SGL to the underlying driver.

17. The data storage system of claim 12 , wherein each per-core thread is one of a set of threads collectively realizing implementing operations of the library component for a DM request, the set of threads including a copy thread, a callback thread, and an I/O request packet (IRP) completion thread, the copy thread being core-affined and being a main execution thread for each DM request, the IRP completion thread also being core-affined and performing IRP completion processing in response to completion of a read IRP and a write IRP initiated by the copy thread, the callback thread performing callback processing in response to a read callback and a write callback from the underlying drivers identifying the source area and destination area of memory for the data copying, both the callback thread and the IRP completion thread generating respective signals to awaken the copy thread to resume operation on a DM request after processing of a callback and IRP completion respectively are completed.

18. The data storage system of claim 17 , wherein the callback thread is not core-affined and executes on a core other than the core on which the copy thread and IRP completion thread execute, and operation of the callback thread for a given data movement request includes (1) identifying which core the data movement request is assigned to, and (2) generating the signal to the copy thread affined to the core to which the data movement request is assigned.

19. The data storage system of claim 17 , wherein the callback processing of the callback thread includes:

(1) storing a source scatter-gather list (SGL) in association with the data movement request;

(2) for a write callback, sending a signal to the copy thread to initiate data copying from the source area to the destination area; and

(3) for a read callback, prepare a write IRP and send it to the underlying driver;

and wherein the copy thread responds to the signal by:

(1) determining whether both source and destination SGLs are available; and

(2) in response to both source destination SGLs being available:

performing a memory copy from source SGL to destination SGL;

confirming the destination SGL to the underlying driver;

determining whether all of the source SGL has been copied to a destination SGL, and if not then looping back to beginning to wait for more destination SGLs to be obtained; and

if all of the source SGL has been copied to a destination SGL, then confirming the source SGL to the underlying driver.

20. The data storage system of claim 17 , wherein the IRP completion processing of the IRP completion thread includes:

for a write IRP completion:

(1) determining whether all the data in a source scatter-gather list (SGL) has been copied, and if not then preparing another write IRP and sending it to the underlying driver;

(2) if all the data in a source scatter-gather list (SGL) has been copied, then sending the signal the copy thread to await a read IRP completion; and

for a read IRP completion:

(1) determining whether all the data of the current data movement request has been copied, and if not then preparing another read IRP and sending it to the underlying driver; and

(2) if all the data of the current data movement request has been copied, then sending the signal the copy thread to complete the processing of the data movement request;

and wherein the copy thread responds to the signal by:

(1) determining whether both the read IRP and write IRP are complete for a sub-range of the data movement request, and if not then awaiting completion of both the read IRP and write IRP;

(2) if both the read IRP and write IRP are complete for the sub-range of the data movement request, then determining whether the processing for the entire data movement request has been performed, and if not then initiating additional device read and copy operations to complete the data movement request.