Employing multiple queueing structures within a userspace storage driver to increase speed
View Patent ↗Performance in multi-core data storage systems is increased while allowing for portability and fast failover in the event of a failure of a driver stack by a data storage system employing several queues to reduce lock contention. Queueing is performed with two levels of several queues each within a userspace scheduling driver within a userspace container. The userspace scheduling driver may dequeue into a userspace management driver that communicates with a kernel-based hardware driver by way of a kernel helper driver. An apparatus, system, and computer program product for performing a similar method are also provided.
1. A method of processing storage requests directed to a storage device of a computing device having a plurality of processing cores, the method comprising:
enqueuing, by a first storage driver operating within userspace of the computing device, storage requests initiated by a first core of the computing device onto a first userspace queue, the first userspace queue being dedicated to storage requests from the first core;
enqueuing, by the first storage driver operating within userspace, storage requests initiated by a second core of the computing device onto a second userspace queue, the second userspace queue being dedicated to storage requests from the second core;
transferring, by the first storage driver operating within userspace, storage requests from the first userspace queue and the second userspace queue to a set of userspace dispatch queues, the first userspace queue and the second userspace queue not belonging to the set of userspace dispatch queues;
sending, by the first storage driver operating within userspace, storage requests from the set of userspace dispatch queues to a second storage driver operating within userspace of the computing device; and
sending, by the second storage driver operating within userspace, by way of a kernel helper function, the storage requests received from the first storage driver to a hardware device driver for the storage device for performance by the storage device, the hardware device driver for the storage device operating within a kernel of the computing device.
2. The method of claim 1
wherein the storage device is configured to process a plurality of storage requests concurrently;
wherein the set of userspace dispatch queues includes a plurality of userspace dispatch queues; and
wherein transferring storage requests from the first queue and the second queue to the set of userspace dispatch queues includes:
enqueuing storage requests from the first userspace queue onto a first userspace dispatch queue of the set of userspace dispatch queues; and
enqueuing storage requests from the second userspace queue onto a second userspace dispatch queue of the set of userspace dispatch queues.
3. The method of claim 2 wherein the method further comprises:
enqueuing, by the first storage driver operating within userspace, storage requests initiated by a third core of the computing device onto a third userspace queue, the third userspace queue being dedicated to storage requests from the third core; and
transferring, by the first storage driver operating within userspace, storage requests from the third userspace queue to the second userspace dispatch queue for performance by the storage device,
wherein transferring the storage requests from the third userspace queue to the second userspace dispatch queue is interleaved with transferring the storage requests from the second userspace queue to the second userspace dispatch queue.
4. The method of claim 2 wherein a number of the plurality of userspace dispatch queues is equal to a number of the plurality of storage requests that the storage device is configured to process concurrently.
5. The method of claim 2 wherein sending the storage requests from the set of userspace dispatch queues to the second storage driver operating within userspace includes dispatching storage requests from each of the plurality of userspace dispatch queues to the second storage driver operating within userspace in an alternating matter according to a fairness policy.
6. The method of claim 1 wherein sending the storage requests from the set of userspace dispatch queues to the second storage driver operating within userspace includes dispatching storage requests from each of the plurality of userspace dispatch queues without reordering or merging the storage requests.
7. The method of claim 1 wherein the first storage driver operating within userspace performs reordering of storage requests within the first userspace queue and the second userspace queue.
8. The method of claim 7 wherein the first storage driver operating within userspace performs merging of contiguous storage requests within the first userspace queue and the second userspace queue.
9. The method of claim 1 wherein the method further comprises performing load-balancing of storage requests between the first userspace queue and the second userspace queue.
10. The method of claim 1
wherein the storage device is one of a plurality of storage devices of the computing system; and
wherein the second storage driver running within userspace is configured to manage the plurality of storage devices.
11. The method of claim 1 , further comprising communicating, by the kernel helper function, hardware interrupts between the hardware device driver for the storage device operating within the kernel and the second storage driver running within userspace.
12. The method of claim 1 , wherein the first storage driver, the first userspace queue, the second storage driver, the second userspace queue, and the set of userspace dispatch queues operate together within a single userspace container.
13. A computing apparatus for processing storage requests directed to a storage device of the computing apparatus, the computing apparatus comprising processing circuitry having a plurality of processing cores coupled to memory configured to:
enqueue, by a first storage driver operating within userspace of the computing apparatus, storage requests initiated by a first core of the computing apparatus onto a first userspace queue, the first userspace queue being dedicated to storage requests from the first core;
enqueue, by the first storage driver operating within userspace, storage requests initiated by a second core of the computing apparatus onto a second userspace queue, the second userspace queue being dedicated to storage requests from the second core;
transfer, by the first storage driver operating within userspace, storage requests from the first userspace queue and the second userspace queue to a set of userspace dispatch queues, the first userspace queue and the second userspace queue not belonging to the set of userspace dispatch queues;
send, by the first storage driver operating within userspace, storage requests from the set of userspace dispatch queues to a second storage driver operating within userspace of the computing apparatus; and
send, by the second storage driver operating within userspace, by way of a kernel helper function, the storage requests received from the first storage driver to a hardware device driver for the storage device for performance by the storage device, the hardware device driver for the storage device operating within a kernel of the computing apparatus.
14. The computing apparatus of claim 13 , wherein the first storage driver, the first userspace queue, the second storage driver, the second userspace queue, and the set of userspace dispatch queues are configured to operate together within a single userspace container.
15. The computing apparatus of claim 14
wherein the storage device is configured to process a plurality of storage requests concurrently;
wherein the set of userspace dispatch queues includes a plurality of userspace dispatch queues; and
wherein the processing circuitry configured to transfer storage requests from the first queue and the second queue to the set of userspace dispatch queues is further configured to:
enqueue storage requests from the first userspace queue onto a first userspace dispatch queue of the set of userspace dispatch queues; and
enqueue storage requests from the second userspace queue onto a second userspace dispatch queue of the set of userspace dispatch queues.
16. The computing apparatus of claim 14 wherein the processing circuitry configured to send the storage requests from the set of userspace dispatch queues to the second storage driver operating within userspace is further configured to dispatch storage requests from each of the plurality of userspace dispatch queues without reordering or merging the storage requests.
17. A computer program product including a non-transitory, computer-readable storage medium storing instructions which, when executed by the computing device, cause the computing device to perform a method of processing storage requests directed to a storage device of a computing device having a plurality of processing cores, the method comprising:
enqueuing, by a first storage driver operating within userspace of the computing device, storage requests initiated by a first core of the computing device onto a first userspace queue, the first userspace queue being dedicated to storage requests from the first core;
enqueuing, by the first storage driver operating within userspace, storage requests initiated by a second core of the computing device onto a second userspace queue, the second userspace queue being dedicated to storage requests from the second core;
transferring, by the first storage driver operating within userspace, storage requests from the first userspace queue and the second userspace queue to a set of userspace dispatch queues, the first userspace queue and the second userspace queue not belonging to the set of userspace dispatch queues;
sending, by the first storage driver operating within userspace, storage requests from the set of userspace dispatch queues to a second storage driver operating within userspace of the computing device; and
sending, by the second storage driver operating within userspace, by way of a kernel helper function, the storage requests received from the first storage driver to a hardware device driver for the storage device for performance by the storage device, the hardware device driver for the storage device operating within a kernel of the computing device.
18. The computer program product of claim 17 , wherein the first storage driver, the first userspace queue, the second storage driver, the second userspace queue, and the set of userspace dispatch queues operate together within a single userspace container.
19. The computer program product of claim 18
wherein the storage device is configured to process a plurality of storage requests concurrently;
wherein the set of userspace dispatch queues includes a plurality of userspace dispatch queues; and
wherein transferring storage requests from the first queue and the second queue to the set of userspace dispatch queues includes:
enqueuing storage requests from the first userspace queue onto a first userspace dispatch queue of the set of userspace dispatch queues; and
enqueuing storage requests from the second userspace queue onto a second userspace dispatch queue of the set of userspace dispatch queues.
20. The computer program product of claim 18 wherein sending the storage requests from the set of userspace dispatch queues to the second storage driver operating within userspace includes dispatching storage requests from each of the plurality of userspace dispatch queues without reordering or merging the storage requests.