Data storage system with cluster virtual memory on non-cache-coherent cluster interconnect
A data storage system includes compute engines each including a CPU complex, physical memory, interfaces to host computers and physical storage devices, and a fabric interface. A non-cache-coherent fabric interconnects the compute engines as cluster members of a cluster, the fabric supporting a global lock enabling each cluster member to obtain temporary exclusive access to addressable units of non-virtual memory. The CPU complexes implement a global virtual memory (GVM) on top of the non-virtual memory, including (1) a globally shared GVM page table of global page table entries, each identifying the cluster members having a corresponding GVM page mapped, and (2) GVM page management functionality including (i) use of the global lock to obtain exclusive access to the global page table entries for page management operations, and (ii) transfer of underlying data of the pages of the GVM among the cluster members.
1. A data storage system, comprising:
a plurality of compute engines each including a CPU complex, physical memory, front-end interfaces to host computers as initiators of storage I/O operations, back-end interfaces to physical storage devices, and a fabric interface; and
a non-cache-coherent fabric interconnect connected to the respective fabric interfaces of the compute engines to functionally interconnect the compute engines as cluster members of a cluster, the fabric interconnect supporting a global lock enabling each of the cluster members to obtain temporary exclusive access to addressable units of non-virtual memory, at least one cluster member having access to persistent storage for memory page swapping,
wherein the CPU complexes collectively implement a global virtual memory on top of the non-virtual memory, including (1) a globally shared GVM page table of global page table entries, each identifying the cluster members having a corresponding virtual page mapped, and (2) GVM page management functionality including (i) use of the global lock to obtain exclusive access to the global page table entries for page management operations, and (ii) transfer of underlying data of the pages of the global virtual memory among the cluster members.
2. The data storage system of claim 1 , wherein the CPU complexes further collectively implement (3) additional page fault handling mechanisms for GVM addresses using the globally shared GVM page table and GVM page management functionality to provide a shared globally visible address space across compute engines utilizing the fabric interconnect.
3. The data storage system of claim 1 , wherein the fabric interconnect supports the transfer of underlying data of the pages of the global virtual memory among the cluster members using remote direct memory access.
4. The data storage system of claim 1 , wherein each compute engine includes redundant directors having respective CPU complexes and GVM memories interconnected by a local common memory interconnect.
5. The data storage system of claim 4 , wherein the content of the GVMs is mirrored between respective pairs of compute engines.
6. The data storage system of claim 1 , wherein the compute engines are configured and operative to use the global virtual memory to store device metadata visible to multiple compute engines, the device metadata describing logical storage devices defined in the system, both host-visible and internal-only, the metadata being read more frequently than the metadata is modified.
7. The data storage system of claim 1 , wherein each of the global page table entries includes a lock field, a writer field, a readers field, a mirror field, a dirty field, and a statistics field, the lock field identifying a compute engine currently holding a lock on a respective GVM page of the global virtual memory, the writer field identifying a current engine allowed to modify the GVM page, the readers field identifying all engines having read copies of the GVM page, the mirror field identifying a mirror status of the GVM page, the dirty field identifying whether the GVM page is modified and not yet written to a backing store, and the statistics field including statistics on use of the GVM page.
8. The data storage system of claim 1 , wherein the non-virtual memory stores the global page table entries and is realized as emulated shared memory by use of a respective function library and application programming interface (API) in each of the CPU complexes, the API including a write command enabling an application program to write data to a specified section of the non-virtual memory, and a read command returning data stored at a specified address of the non-virtual memory.
9. The data storage system of claim 8 , wherein each compute engine includes redundant directors having respective CPU complexes, and a write command performed in a CPU complex of one director is automatically mirrored to the other director of the compute engine.
10. The data storage system of claim 8 , wherein the application program is responsible for maintaining coherency of data of the non-virtual memory across the directors of the system.
11. A method of operating a data storage system having a plurality of compute engines and a non-cache-coherent fabric interconnect, the compute engines each including a CPU complex, physical memory, front-end interfaces to host computers as initiators of storage I/O operations, back-end interfaces to physical storage devices, and a fabric interface, and the non-cache-coherent fabric interconnect being connected to the respective fabric interfaces of the compute engines to functionally interconnect the compute engines as cluster members of a cluster, the fabric interconnect supporting a global lock enabling each of the cluster members to obtain temporary exclusive access to addressable units of non-virtual memory, at least one cluster member having access to persistent storage for memory page swapping, the method including:
collectively implementing, by the CPU complexes, a global virtual memory on top of the non-virtual memory, the global virtual memory including a globally shared GVM page table of global page table entries, each identifying the cluster members having a corresponding virtual page mapped; and
providing GVM page management functionality including (i) use of the global lock to obtain exclusive access to the global page table entries for page management operations, and (ii) transfer of underlying data of the pages of the global virtual memory among the cluster members.
12. The method of claim 11 , further including, by the CPU complexes, further collectively implementing (3) additional page fault handling mechanisms for GVM addresses using the globally shared GVM page table and GVM page management functionality to provide a shared globally visible address space across compute engines utilizing the fabric interconnect.
13. The method of claim 11 , further including, for the transfer of underlying data of the pages of the global virtual memory among the cluster members, using remote direct memory access.
14. The method of claim 11 , wherein each compute engine includes redundant directors having respective CPU complexes and GVM memories interconnected by a local common memory interconnect.
15. The method of claim 14 , further including mirroring the content of the GVMs between respective pairs of compute engines.
16. The method of claim 11 , further including, by the compute engines, using the global virtual memory to store device metadata visible to multiple compute engines, the device metadata describing logical storage devices defined in the system, both host-visible and internal-only, the metadata being read more frequently than the metadata is modified.
17. The method of claim 11 , wherein each of the global page table entries includes a lock field, a writer field, a readers field, a mirror field, a dirty field, and a statistics field, the lock field identifying a compute engine currently holding a lock on a respective GVM page of the global virtual memory, the writer field identifying a current engine allowed to modify the GVM page, the readers field identifying all engines having read copies of the GVM page, the mirror field identifying a mirror status of the GVM page, the dirty field identifying whether the GVM page is modified and not yet written to a backing store, and the statistics field including statistics on use of the GVM page.