Lock-free raid implementation in multi-queue architecture
Systems and methods (including hardware and software) are disclosed where all common RAID storage levels are implemented for multi-queue hardware by isolating RAID stripes to a single central processing unit (CPU) core affinity. Fixed CPU affinity is used for any piece of data that may be modified. Instead of blocking CPUs that must access or modify a piece of data, the request is efficiently moved to the CPU that owns that data. In this manner the system is completely asynchronous, efficient, and scalable.
1. A system, comprising:
a plurality of central processing unit (CPU) cores; and
a storage device for storing a plurality of data stripes; and
a non-transitory computer readable storage medium, comprising instructions for:
at a first CPU core of the plurality of CPU cores having a first CPU core number:
receiving a client input/output (IO) request having a data address;
determining a stripe number for one of the plurality of data stripes as a function of the data address;
determining a second CPU core number as a function of the stripe number; and
routing the I/O request from the first CPU core to a second CPU core of the plurality of CPU cores having the second CPU core number, wherein routing the I/O request from the first CPU core to the second CPU core comprises:
forming, at the first CPU core, a first message associated with the I/O request; and
the first CPU core placing the message on a first producer-consumer queue associated with the second CPU core; and
at the second CPU core having the second CPU core number:
obtaining the first message from the first producer-consumer queue based on the determination that the first message is on the first producer-consumer queue and performing the request on the determined stripe number; and
routing a request completion from the second CPU core to the first CPU core, wherein routing the request completion from the second CPU core to the first CPU core comprises:
forming, at the second CPU core, a second message associated with request completion; and
the second CPU core placing the second message on a second producer-consumer queue associated with the first CPU core.
2. The system of claim 1 , wherein the first producer-consumer queue is specific to messages from the first CPU core to the second CPU core.
3. The system of claim 2 , wherein system comprises a plurality of CPU sockets including the plurality of CPU cores, the first CPU core is in a first CPU socket of the plurality of CPU sockets, and the second CPU core is in a second CPU socket of the plurality of CPU sockets.
4. The system of claim 3 , wherein the first CPU socket has first associated memory and the second CPU socket has second associated memory and the first producer-consumer queue is stored in the second memory associated with the second CPU socket.
5. The system of claim 4 , wherein routing the I/O request to the second CPU core further comprises incrementing, by the first CPU core, a first producer index associated with the first producer-consumer queue.
6. The system of claim 5 , wherein the non-transitory computer readable storage medium comprises instructions for, at the second CPU core: determining that the first message is on the first producer-consumer queue based on a difference between a first consumer index associated with the first producer-consumer queue and the first producer index associated with the first producer-consume queue.
7. The system of claim 6 , wherein the first producer index and the first consumer index are stored in the second memory associated with the second CPU socket.
8. The system of claim 2 , wherein the second producer-consumer queue is specific to messages from the second CPU core to the first CPU core.
9. A method for lock-free RAID implementation, comprising:
receiving, at a first central processing unit (CPU) core having a first CPU core number, a client input/output (IO) request having a data address;
determining, at the first CPU core, a stripe number as a function of the data address;
determining, at the first CPU core, a second CPU core number as a function of the stripe number; and
routing the I/O request from the first CPU core to a second CPU core having the second CPU core number, wherein routing the I/O from the first CPU core to the second CPU core comprises:
forming, at the first CPU core, a first message associated with the I/O request; and
the first CPU core placing the message on a first producer-consumer queue associated with the second CPU core;
obtaining, at the second CPU core having the second CPU core number, the first message from the first producer-consumer queue based on the determination that the first message is on the first producer-consumer queue:
performing, at the second CPU core, the request on the determined stripe number:
routing a request completion from the second CPU core to the first CPU core, wherein routing the request completion from the second CPU core to the first CPU core comprises:
forming, at the second CPU core, a second message associated with request completion; and
the second CPU core placing the second message on a second producer-consumer queue associated with the first CPU core.
10. The method of claim 9 , wherein the first producer-consumer queue is specific to messages from the first CPU core to the second CPU core.
11. The method of claim 10 , wherein the first CPU core is in a first CPU socket and the second CPU core is in a second CPU socket.
12. The method of claim 11 , wherein the first CPU socket has first associated memory and the second CPU socket has second associated memory and the first producer-consumer queue is in the second memory associated with the second CPU socket.
13. The method of claim 12 , wherein routing the I/O request to the second CPU core further comprises incrementing, by the first CPU core, a first producer index associated with the first producer-consumer queue.
14. The method of claim 13 , further comprising, determining by the second CPU core that the first message is on the first producer-consumer queue based on a difference between a first consumer index associated with the first producer-consumer queue and the first producer index associated with the first producer-consume queue.
15. The method of claim 14 , wherein the first producer index and the first consumer index are stored in the second memory associated with the second CPU socket.
16. The method of claim 10 , wherein the second producer-consumer queue is specific to messages from the second CPU core to the first CPU core.