Failure fault tolerance in distributed memory systems
Disclosed in some examples are methods, systems, devices, and architectures which provide for techniques for memory device and memory fabric redundancy within distributed memory systems. In some examples, two memory devices are paired and each stores a same set of data such that writes to the memory devices are duplicated and reads may be satisfied from either device. In some examples, a memory processing unit (MPU) may be incorporated into the memory architecture to support these paired memory devices. The MPU may be placed between the host and a multi-planed memory fabric which connects to multi-ported CXL memory devices. In some examples, the MPU may also enable the use of alternative fabric links. That is, if a memory fabric link between the MPU and a memory device is unavailable, an alternative link may be utilized to restore connectivity to a memory device.
1 . A method for failure fault tolerance in a distributed memory system, the method comprising:
at a memory processing unit connected across a host interface to a host and a memory fabric interface to a plurality of memory devices:
receiving, from the host across the host interface, a read request for a first memory location;
identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices comprising a first memory device and a second memory device;
determining a primary memory device as either the first or second memory device and a secondary memory device as the other of the first or second memory device;
transmitting the read request to the primary memory device over the memory fabric interface;
failing to receive data in the first memory location within a prespecified threshold period of time;
responsive to failing to receive the data in the first memory location within the prespecified threshold period of time, sending the read request to the secondary memory device over the memory fabric interface;
subsequently to sending the read request to the secondary memory device:
receiving, from the host across the host interface, a write request;
duplicating the write request to create a second write request; and
transmitting the write request to the primary memory device over the memory fabric interface and the second write request to the secondary memory device over the memory fabric interface.
2 . The method of claim 1 , wherein determining the primary memory device as either the first or second memory device comprises utilizing one or more bits in an address of the first memory location.
3 . The method of claim 1 , further comprising:
setting a timer prior to transmitting the read request to the primary memory device, the timer less than the prespecified threshold period of time;
receiving a timer expiry prior to receiving a response from the primary memory device; and
responsive to receiving the timer expiry prior to receiving the response from the primary memory device, retransmitting the read request to the primary memory device over an alternative memory fabric link.
4 . The method of claim 3 , further comprising:
setting a second timer prior to retransmitting the read request to the primary memory device over the alternative memory fabric link;
receiving a second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link; and
wherein the sending the read request to the secondary memory device over the memory fabric interface is performed responsive to receiving the second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link.
5 . The method of claim 1 , further comprising:
determining that the first memory device is to be replaced;
responsive to determining that the first memory device is to be replaced:
updating a read target data structure to identify that the first memory device is out of service;
updating a write target data structure to identify that the first memory device is out of service; and
subsequent to updating the read target data structure and write target data structure, servicing subsequent read or write requests for addresses corresponding to the pair of memory devices using the second memory device only by sending read or write requests only to the second memory device.
6 . The method of claim 5 , further comprising:
identifying that a replacement memory device for the first memory device has been installed;
updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices;
reading the data from the second memory device;
setting the data from the second memory device as modified;
flushing the data from the second memory device; and
updating the read target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices.
7 . A memory processing unit configured to be connected across a host interface to a host and a memory fabric interface to a plurality of memory devices, for providing failure fault tolerance in a distributed memory system, the memory processing unit comprising:
a processor, configured to perform operations comprising:
receiving from a host across a host interface, a read request for a first memory location;
identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices comprising a first memory device and a second memory device;
determining a primary memory device as either the first or second memory device and a secondary memory device as the other of the first or second memory device;
transmitting the read request to the primary memory device over a memory fabric interface;
failing to receive data in the first memory location within a prespecified threshold period of time;
responsive to failing to receive the data in the first memory location within the prespecified threshold period of time, sending the read request to the secondary memory device over the memory fabric interface;
subsequently to sending the read request to the secondary memory device:
receiving, from the host across the host interface, a write request;
duplicating the write request to create a second write request; and
transmitting the write request to the primary memory device over the memory fabric interface and the second write request to the secondary memory device over the memory fabric interface.
8 . The memory processing unit of claim 7 , wherein the operations of determining the primary memory device as either the first or second memory device comprises utilizing one or more bits in an address of the first memory location.
9 . The memory processing unit of claim 7 , wherein the operations further comprise:
setting a timer prior to transmitting the read request to the primary memory device, the timer less than the prespecified threshold period of time;
receiving a timer expiry prior to receiving a response from the primary memory device; and
responsive to receiving the timer expiry prior to receiving the response from the primary memory device, retransmitting the read request to the primary memory device over an alternative memory fabric link.
10 . The memory processing unit of claim 9 , wherein the operations further comprise:
setting a second timer prior to retransmitting the read request to the primary memory device over the alternative memory fabric link;
receiving a second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link; and
wherein the sending the read request to the secondary memory device over the memory fabric interface is performed responsive to receiving the second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link.
11 . The memory processing unit of claim 7 , wherein the operations further comprise:
determining that the first memory device is to be replaced;
responsive to determining that the first memory device is to be replaced:
updating a read target data structure to identify that the first memory device is out of service;
updating a write target data structure to identify that the first memory device is out of service; and
subsequent to updating the read target data structure and write target data structure, servicing subsequent read or write requests for addresses corresponding to the pair of memory devices using the second memory device only by sending read or write requests only to the second memory device.
12 . The memory processing unit of claim 11 , wherein the operations further comprise:
identifying that a replacement memory device for the first memory device has been installed;
updating the write target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices;
reading the data from the second memory device;
setting the data from the second memory device as modified;
flushing the data from the second memory device; and
updating the read target data structure to identify that the replacement memory device and the second memory device are a pair of memory devices.
13 . A non-transitory machine-readable medium storing instructions, for failure fault tolerance in a distributed memory system, the instructions, when executed by a memory processing unit connected across a host interface to a host and a memory fabric interface to a plurality of memory devices, cause the memory processing unit to perform operations comprising:
receiving, from the host across the host interface, a read request for a first memory location;
identifying that the first memory location is a mirrored memory location corresponding to a pair of memory devices comprising a first memory device and a second memory device;
determining a primary memory device as either the first or second memory device and a secondary memory device as the other of the first or second memory device;
transmitting the read request to the primary memory device over the memory fabric interface;
failing to receive data in the first memory location within a prespecified threshold period of time;
responsive to failing to receive the data in the first memory location within the prespecified threshold period of time, sending the read request to the secondary memory device over the memory fabric interface;
subsequently to sending the read request to the secondary memory device:
receiving, from the host across the host interface, a write request;
duplicating the write request to create a second write request; and
transmitting the write request to the primary memory device over the memory fabric interface and the second write request to the secondary memory device over the memory fabric interface.
14 . The non-transitory machine-readable medium of claim 13 , wherein the operations of determining the primary memory device as either the first or second memory device comprises utilizing one or more bits in an address of the first memory location.
15 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise:
setting a timer prior to transmitting the read request to the primary memory device, the timer less than the prespecified threshold period of time;
receiving a timer expiry prior to receiving a response from the primary memory device; and
responsive to receiving the timer expiry prior to receiving the response from the primary memory device, retransmitting the read request to the primary memory device over an alternative memory fabric link.
16 . The non-transitory machine-readable medium of claim 15 , wherein the operations further comprise:
setting a second timer prior to retransmitting the read request to the primary memory device over the alternative memory fabric link;
receiving a second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link; and
wherein the sending the read request to the secondary memory device over the memory fabric interface is performed responsive to receiving the second timer expiry prior to receiving the response from the primary memory device over the alternative memory fabric link.
17 . The non-transitory machine-readable medium of claim 13 , wherein the operations further comprise:
determining that the first memory device is to be replaced;
responsive to determining that the first memory device is to be replaced:
updating a read target data structure to identify that the first memory device is out of service;
updating a write target data structure to identify that the first memory device is out of service; and
subsequent to updating the read target data structure and write target data structure, servicing subsequent read or write requests for addresses corresponding to the pair of memory devices using the second memory device only by sending read or write requests only to the second memory device.