Method of submitting work to fabric attached memory

1 . A method performed by a distributed computing system that includes one or more host nodes and fabric attached memory (FAM) devices connected to the one or more host nodes by a fabric interconnect, the method comprising:

receiving a work packet via the fabric interconnect at a first command manager (CM) of a memory controller of a first FAM device from a second CM of a second FAM device, wherein the work packet includes a memory access to be performed by a FAM computing resource local to the first FAM device;

receiving, in a CM response queue of the first CM, indication of completion of the work packet from the FAM computing resource of the first FAM device; and

sending, by the first CM, a response packet to a CM response queue of the second CM.

2 . The method of claim 1 , including:

determining, by the first CM, when the CM response queue of the second CM has space available; and

writing, by the first CM, a completion response message to the CM response queue of the second CM when the work packet is completed and the CM response queue of the second CM has space available.

3 . The method of claim 2 , including:

updating, by the first CM, a write index pointer when writing the completion response message to the CM response queue of the second CM;

sending, by the second CM, a read index pointer to the first CM when reading the response from the CM response queue; and

wherein the determining when the CM response queue of the second CM has space available includes the first CM determining that the CM response queue of the second CM has space available using the write index pointer and the read index pointer.

4 . The method of claim 1 , including the second CM sending the work packet to the first CM using a compute express link (CXL) memory write transaction.

5 . The method of claim 4 , including:

sending, by the first CM, no response to the second CM when receiving the work packet; and

sending, by the first CM, the response packet to the CM response queue of the second CM when the work packet is completed.

6 . The method of claim 1 , including:

placing, by the first CM, the work packet in a CM work queue for the FAM computing resource when space is available in the CM work queue for the FAM computing resource; and

placing, by the first CM when the CM work queue for the FAM computing resource is full, the work packet in a destination work queue according to an address included in the work packet, wherein the destination queue is implemented in a memory array of the first FAM device external to the memory controller of the first FAM device.

7 . The method of claim 6 , including:

reading, by the first CM, the work packet from the destination queue and writing the work packet to the CM work queue when becomes available in the CM work queue.

8 . The method of claim 1 , including producing the work packet using a FAM tenant of the second FAM device.

9 . A fabric attached memory (FAM) device of a distributed computing system, the FAM device comprising:

a memory including a memory array of memory cells;

a fabric interface to a fabric interconnect of the distributed computing system;

at least one FAM computing resource located on a memory-side of the fabric interface; and

a memory controller operatively coupled to the memory and the at least one FAM computing resource, wherein the memory controller includes a command manager (CM); and wherein the CM includes:

a CM work queue local to the memory controller and configured to store work packets performable by the at least one FAM computing resource;

a CM response queue configured to receive an indication of completion of the work packet from the FAM computing resource; and

CM logic circuitry configured to:

receive a work packet via the fabric interconnect from another CM of another FAM device, wherein the work packet includes a memory access to be performed by the at least one FAM computing resource;

receive indication of completion of the work packet from the FAM computing resource in the CM response queue; and

send a response packet to a CM response queue of the other CM in response to receiving receive indication of completion of the work packet.

10 . The FAM device of claim 9 , wherein the CM logic circuitry is further configured to:

determine when the CM response queue of the other CM has space available; and

write a completion response message to the CM response queue of the other CM when the work packet is completed and the CM response queue of the other CM has space available.

11 . The FAM device of claim 10 , wherein the CM logic circuitry is further configured to:

update a write index pointer when sending the response packet to the CM response queue of the other CM;

receive a read index pointer of the CM response queue of the other CM; and

compare the write index of the CM response queue of the other CM to the read index of the CM response queue of the other CM to determine if the other CM response queue of the other CM has space available.

12 . The FAM device of claim 9 , wherein the CM logic circuitry is configured to receive the work packet from the other CM according to a compute express link (CXL) memory write transaction.

13 . The FAM device of claim 12 , wherein the CM logic circuitry is configured to:

send no response to the other CM when receiving the work packet; and

send the response packet to the CM response queue of the other CM when the work packet is completed.

14 . The FAM device of claim 9 , wherein the CM logic circuitry is configured to:

store the work packet in a CM work queue for the FAM computing resource when space is available in the CM work queue for the FAM computing resource; and

store the work packet in a destination work queue according to an address included in the work packet when the CM work queue for the FAM computing resource is full, wherein the destination queue is implemented in a memory array external to the memory controller.

15 . The FAM device of claim 14 , wherein the CM logic circuitry is configured to read the work packet from the destination queue and write the work packet to the CM work queue when becomes available in the CM work queue.

16 . The FAM device of claim 9 , wherein the CM logic circuitry is configured to store the work packet in a CM work queue for a FAM tenant of the other FAM device.

17 . A non-transitory computer readable storage medium including instructions that, when performed by processing circuitry of a memory controller of a fabric attached memory (FAM) device, cause the FAM device to perform acts comprising:

receiving a work packet via a fabric interconnect at a first command manager (CM) of a memory controller of a first FAM device from a second CM of a second FAM device, wherein the work packet includes a memory access to be performed by a FAM computing resource local to the first FAM device;

receiving, in a CM response queue of the first CM, indication of completion of the work packet from the FAM computing resource of the first FAM device; and

sending, by the first CM, a response packet to a CM response queue of the second CM.

18 . The non-transitory computer readable storage medium of claim 17 , further including instructions that cause the FAM device to perform acts comprising:

determining when the CM response queue of the second CM has space available; and

writing a completion response message to the CM response queue of the second CM when the work packet is completed and the CM response queue of the second CM has space available.

19 . The non-transitory computer readable storage medium of claim 17 , further including instructions that cause the FAM device to perform acts comprising:

receiving the work packet according to a compute express link (CXL) memory write transaction;

sending no response to the second CM when receiving the work packet; and

sending the response packet to the CM response queue of the second CM when the work packet is completed.

20 . The non-transitory computer readable storage medium of claim 17 , further including instructions that cause the FAM device to perform acts comprising:

placing the work packet in a CM work queue for the FAM computing resource when space is available in the CM work queue for the FAM computing resource; and

placing the work packet in a destination work queue of the first FAM device according to an address included in the work packet when the CM work queue for the FAM computing resource is full, wherein the destination queue is external to the memory controller of the first FAM device.