Storage system with multicast DMA and unified address space
A system and method for clients, a control module, and storage modules to participate in a unified address space in order to and read and write data efficiently using direct-memory access. The method for reading data includes determining a first location in a first memory to write a first copy of the data, a second location in a second memory to write a second copy of the data, where the first memory is located in a first storage module including a first persistent storage and the second memory is located in a second storage module including a second persistent storage. The method further includes programming a direct memory access engine to read the data from client memory and issue a first write request to a multicast address, where the first location, the second location, and a third location are associated with the multicast address.
1. A method for writing data, comprising:
receiving, from a client, a request to write data, wherein the data is located in client memory;
making a first determination to store a first copy of the data in a first persistent storage in a first storage module and to store a second copy of the data in a second persistent storage in a second storage module;
based on the first determination selecting, by a storage appliance, a first location in a first memory to write the first copy of the data, a second location in a second memory to write the second copy of the data, wherein the first memory is located in the first storage module and the second memory is located in the second storage module, wherein the storage appliance comprises the first storage module and the second storage module;
programming a direct memory access (DMA) engine to read the data from the client memory and issue a first write request to a multicast address, wherein the first location, the second location, and a third location are associated with the multicast address;
after programming:
receiving, from the DMA engine, the first request at a switch in a switch fabric in the storage appliance;
in response to receiving the first request, generating, in the switch, a second request to write the first copy to the first location, a third request to write the second copy to the second location, and a fourth request to write the third copy to the third location, wherein the third location is in a third memory in a control module in the storage appliance;
wherein the first copy of the data is written to the first location in response to the second request;
wherein the second copy of the data is written to the second location in response to the third request;
wherein the third copy of the data is written to the third location in response to the fourth request;
wherein the DMA engine reads the data from the client memory;
wherein the first memory, the second memory, the third memory, and, at least a portion of the client memory are part of a storage fabric accessible by a unified address space.
2. The method of claim 1 , further comprising:
after the first copy of the data is written to the first location, the second copy of the data is written to the second location, and the third copy of the data is written to the third location:
notifying the client that the request to write the data was completed.
3. The method of claim 2 , further comprising:
after notifying the client:
writing the first copy of the data to the first persistent storage; and
removing, after writing, the first copy of the data from the first memory.
4. The method of claim 3 , wherein a processor in the storage appliance initiates the writing of the first copy of the data to the first persistent storage.
5. The method of claim 3 , wherein the first persistent storage is a solid state memory module.
6. The method of claim 5 , wherein the solid state memory module comprises a plurality of NAND dies.
7. The method of claim 3 , further comprising:
after notifying the client:
removing the second copy of the data from the second memory, and
removing the third copy of the data from the third memory.
8. The method of claim 1 , wherein the request to write the data is received from the client via a client switch, wherein the client communicates with the client switch using Peripheral Component Interconnect Express (PCIe), and the storage appliance communicates with the client switch using PCIe.
9. The method of claim 8 ,
wherein the client is a first root complex,
wherein the storage appliance comprises a processor, wherein the processor is a second root complex,
wherein the client switch comprises a non-transparent bridge,
wherein the at least one copy of the data is transmitted across the non-transparent bridge to the storage appliance.
10. The method of claim 8 , wherein the DMA engine is located in the client switch.
11. The method of claim 1 , wherein a processor in the storage appliance programs the DMA engine.
12. The method of claim 1 , wherein the first location, the second location, and the third location are part of a multicast group implemented by the DMA engine.
13. The method of claim 1 , wherein the first memory comprises vaulted memory, wherein the first location is in the vaulted memory, and, wherein contents of the vaulted memory are automatically written to the first persistent storage in the event of a power failure in the first storage module.
14. A non-transitory computer readable medium comprising instructions which when executed perform a method, the method comprising:
receiving, from a client, a request to write data, wherein the data is located in client memory;
making a first determination to store a first copy of the data in a first persistent storage in a first storage module and to store a second copy of the data in a second persistent storage in a second storage module;
based on the first determination selecting, by a storage appliance, a first location in a first memory to write the first copy of the data, a second location in a second memory to write the second copy of the data, wherein the first memory is located in the first storage module and the second memory is located in the second storage module, wherein the storage appliance comprises the first storage module and the second storage module;
programming a direct memory access (DMA) engine to read the data from the client memory and issue a first write request to a multicast address, wherein the first location, the second location, and a third location are associated with the multicast address;
after programming:
receiving, from the DMA engine, the first request at a switch in a switch fabric in the storage appliance;
in response to receiving the first request, generating, in the switch, a second request to write the first copy to the first location, a third request to write the second copy to the second location, and a fourth request to write the third copy to the third location, wherein the third location is in a third memory in a control module in the storage appliance;
wherein the first copy of the data is written to the first location in response to the second request;
wherein the second copy of the data is written to the second location in response to the third request;
wherein the third copy of the data is written to the third location in response to the fourth request;
wherein the DMA engine reads the data from the client memory;
wherein the first memory, the second memory, the third memory, and, at least a portion of the client memory are part of a storage fabric accessible by a unified address space.
15. The non-transitory computer readable medium of claim 14 , further comprising:
after the first copy of the data is written to the first location, the second copy of the data is written to the second location, and the third copy of the data is written to the third location:
notifying the client that the request to write the data was completed.
16. The non-transitory computer readable medium of claim 15 , further comprising:
after notifying the client:
writing the first copy of the data to the first persistent storage; and
removing, after writing, the first copy of the data from the first memory.
17. The non-transitory computer readable medium of claim 15 , further comprising:
after notifying the client:
removing the second copy of the data from the second memory, and
removing the third copy of the data from the third memory.
18. The non-transitory computer readable medium of claim 14 , wherein the request to write the data is received from the client via a client switch, wherein the client communicates with the client switch using Peripheral Component Interconnect Express (PCIe), and the storage appliance communicates with the client switch using PCIe.
19. The non-transitory computer readable medium of claim 14 , wherein the request to write the data is received from the client via a client switch, wherein the client communicates with the client switch using Peripheral Component Interconnect Express (PCIe), and the storage appliance communicates with the client switch using PCIe.
20. The non-transitory computer readable medium of claim 19 ,
wherein the client is a first root complex,
wherein the storage appliance comprises a processor, wherein the processor is a second root complex,
wherein the client switch comprises a non-transparent bridge,
wherein the at least one copy of the data is transmitted across the non-transparent bridge to the storage appliance.
21. The non-transitory computer readable medium of claim 14 , wherein the first location, the second location, and the third location are part of a multicast group implemented by the DMA engine.