Collective operation using a network-attached memory
In some examples, a processor receives a first request to allocate a memory region for a collective operation by process entities in a plurality of computer nodes. In response to the first request, the processor creates a virtual address for the memory region and allocates the memory region in a network-attached memory coupled to the plurality of computer nodes over a network. The processor correlates the virtual address to an address of the memory region in mapping information. The processor identifies the memory region in the network-attached memory by obtaining the address of the memory region from the mapping information using the virtual address in a second request. In response to the second request, the processor performs the collective operation.
1 . A method comprising:
receiving, by a processor of a first computer node, a first request to allocate a memory region of fabric attached memory having a memory server external to the first computer node, wherein the allocation of memory is for a collective operation with the memory region as an intermediary mechanism for exchanging data between a plurality of computer nodes in the collective operation, wherein the collective operation comprises a pattern of communications among process entities in the plurality of computer nodes that includes the first computer node, and wherein the fabric attached memory includes a memory server separate from the plurality of computer nodes;
in response the first request, creating, by the processor, a virtual address for the memory region and allocating the memory region coupled to the plurality of computer nodes over a network;
correlating, by the processor, the virtual address to an address of the memory region in mapping information stored in the first computer node;
receiving, by the processor, a second request to perform the collective operation, the second request comprising the virtual address;
identifying, by the processor, the memory region by obtaining the address of the memory region from the mapping information using the virtual address in the second request; and
in response to the second request, performing, by the processor, the collective operation comprising writing a first data unit of the first computer node to a first segment of the memory region and accessing a second data unit written by a second computer node from a second segment of the memory region, wherein performing the collective operation comprises exchanging data between the first and second computer nodes via the memory region at least partially instead of sending direct messages between the first and second computer nodes as part of the collective operation.
2 . The method of claim 1 , further comprising:
in response to the second request, performing, by a processor of the second computer node, the collective operation by writing the second data unit to the second segment of the memory region and accessing the first data unit from the first segment of the memory region.
3 . The method of claim 1 , further comprising:
performing, by a plurality of memory servers managing the allocation of the memory region and access of data in the fabric attached memory, computations of a further collective operation on a plurality of data units stored in the fabric attached memory.
4 . The method of claim 3 , further comprising:
retrieving, by the plurality of computer nodes, a result of the computations from the fabric attached memory.
5 . The method of claim 3 , wherein the performing of the computations by the plurality of memory servers is in response to an offload request from the first computer node.
6 . The method of claim 1 , wherein the allocating of the memory region in the fabric attached memory comprises allocating the first segment of the memory region to a first process entity in the first computer node, and allocating the second segment of the memory region to a second process entity in the second computer node.
7 . The method of claim 6 , wherein each of the first process entity and the second process entity is assigned a rank number, the allocating of the first segment of the memory region to the first process entity comprises allocating the first segment starting at a first offset based on the rank number of the first process entity, and the allocating of the second segment of the memory region to the second process entity comprises allocating the second segment starting at a second offset based on the rank number of the second process entity.
8 . The method of claim 1 , wherein the allocating of the memory region in the fabric attached memory in response the first request is responsive to the first request comprising first configuration information having a first value indicating use of the fabric attached memory for the collective operation.
9 . The method of claim 8 , further comprising:
receiving, by the processor of the first computer node, a third request to allocate a further memory region for a further collective operation by the plurality of computer nodes, the third request comprising second configuration information having a second value indicating use of a local memory of the first computer node for the further collective operation; and
in response to the third request, allocating, by the processor, the further memory region in the local memory of the first computer node.
10 . The method of claim 1 , wherein the performing, by the processor, of the collective operation further comprises accessing a third data unit written by a third computer node from a third segment of the memory region.
11 . The method of claim 1 , wherein the collective operation comprises a Message Passing Interface (MPI) collective operation.
12 . The method of claim 1 , wherein the mapping information correlates the virtual address to a descriptor that comprises addresses of memory segments in the memory region in the fabric attached memory, the memory segments allocated to the process entities in the plurality of computer nodes.
13 . The method of claim 1 , comprising:
confirming that the fabric attached memory is used for the collective operation in response to successfully performing a lookup of the mapping information using the virtual address.
14 . The method of claim 1 , wherein the collective operation is performed by the process entities executing in the plurality of computer nodes, the first request is issued by a first process entity of the process entities, and the method comprises:
retrieving, by a second process entity of the process entities, the mapping information; and
using, by the second process entity, the mapping information in the collective operation.
15 . A system comprising:
a fabric attached memory; and
a plurality of computer nodes coupled to the fabric attached memory over a network, the plurality of computer nodes comprising a first computer node and a second computer node, the fabric attached memory having a memory server external to the first computer node, and the first computer node comprising a processor configured to:
receive a first request to allocate a memory region of the fabric attached memory, wherein the allocation of memory is for a collective operation with the memory region as an intermediary mechanism for exchanging data between the plurality of computer nodes in the collective operation, wherein the collective operation comprises a pattern of communications among process entities in the plurality of computer nodes, the first request comprising a first value indicating use of the fabric attached memory for the collective operation;
in response to the first request comprising the first value, create a virtual address for the memory region and allocate the memory region in the fabric attached memory;
add, to mapping information, an entry correlating the virtual address to an address of the memory region in the fabric attached memory;
receive a second request to perform the collective operation, the second request comprising the virtual address;
identify the memory region in the fabric attached memory by obtaining the address of the memory region from the mapping information using the virtual address in the second request; and
in response to the second request, perform the collective operation by writing a first data unit of the first computer node to a first segment of the memory region and accessing a second data unit written by the second computer node from a second segment of the memory region, wherein performing the collective operation comprises exchanging data between the first and second computer nodes via the memory region at least partially instead of sending direct messages between the first and second computer nodes as part of the collective operation.
16 . The system of claim 15 , wherein the fabric attached memory comprises a plurality of memory servers configured to execute computations of a further collective operation offloaded from the plurality of computer nodes to the plurality of memory servers.
17 . The system of claim 16 , wherein the plurality of memory servers are configured to store a result of the computations in the memory region, and the plurality of computer nodes are configured to retrieve the result from the memory region.
18 . The system of claim 15 , wherein the allocating of the memory region in the fabric attached memory comprises allocating the first segment of the memory region to a first process entity in the first computer node, and allocating the second segment of the memory region to a second process entity in the second computer node, and
wherein each of the first process entity and the second process entity is assigned a rank number, the allocating of the first segment of the memory region to the first process entity comprises allocating the first segment starting at a first offset based on the rank number of the first process entity, and the allocating of the second segment of the memory region to the second process entity comprises allocating the second segment starting at a second offset based on the rank number of the second process entity.
19 . A non-transitory machine-readable storage medium comprising instructions that upon execution cause a system to:
receive, at a first computer node, a first request to allocate a memory region of a fabric attached memory having a memory server external to the first computer node, wherein the allocation of memory is for a collective operation with the memory region as an intermediary mechanism for exchanging data between a plurality of computer nodes in the collective operation, wherein the collective operation comprises a pattern of communications among process entities in the plurality of computer nodes that includes the first computer node;
in response the first request, create a virtual address for the memory region and allocate the memory region coupled to the plurality of computer nodes over a network;
correlate the virtual address to an address of the memory region in mapping information stored in the first computer node;
receive a second request to perform the collective operation, the second request comprising the virtual address;
identify the memory region by obtaining the address of the memory region from the mapping information using the virtual address in the second request; and
in response to the second request, perform the collective operation by writing a first data unit of the first computer node to a first segment of the memory region and accessing a second data unit written by a second computer node from a second segment of the memory region, wherein performing the collective operation comprises exchanging data between the first and second computer nodes via the memory region at least partially instead of sending direct messages between the first and second computer nodes as part of the collective operation.
20 . The non-transitory machine-readable storage medium of claim 19 , wherein the collective operation is performed by P process entities in the plurality of computer nodes, and the collective operation involves 2*(P−1) communications of data units that complete the collective operation, and wherein P is greater than or equal to 2.