Systems and methods for interconnecting GPU accelerated compute nodes of an information handling system
An information handling system includes first and second compute nodes, each compute node including a central processing unit (CPU), a computational accelerator (CAC). An inter-accelerator transport (IAT) interface of each node connects to an IAT transport to provide an external interconnect, directly coupling first node GPUs with second node GPUs, for inter-node GPU-to-GPU (GtG) data traffic. Inter-node adapters on each node connect to an inter-node transport (INT) to provide an external interconnect coupling the GPUs of one node to the CPU/root of the other node for carrying inter-node non-GtG data traffic. Interconnects carrying non-GtG traffic, including the INT, may be ×16 PCIe 3.0 or later links while interconnects carrying GtG traffic, including the IAT interconnects, may be implemented as greater-than-PCIe (GTP) interconnects where GTP interconnects have a data transfer rate exceeding comparable PCIe data transfer rates, e.g., 16 GB/s per ×16 PCIE 3.0.
1. An information handling system, comprising:
a first compute node, wherein the first compute node includes:
a central processing unit (CPU);
a first computational accelerator (CAC) comprising a plurality of graphics processing units (GPUs);
a node switch;
a first inter-accelerator transport (TAT) interface configured to receive an IAT comprising one or more IAT interconnects, wherein each IAT interconnect is configured to carry inter-node GPU-to-GPU (GtG) traffic, comprising data traffic between a GPU in the first CAC and a GPU in a second CAC, wherein the second CAC comprises a CAC of a second compute node; and
compute node interconnects, wherein the compute node interconnects include:
node switch interconnects, wherein the node switch interconnects include:
an interconnect coupling the CPU and the node switch; and
one or more switch-to-GPU (StG) interconnects coupling the node switch and one or more of the plurality of GPUs; and
a plurality of CAC interconnects, including:
one or more (GtI) interconnects, wherein each of the one or more GtI interconnects connects a GPU in the first CAC to the IAT interface; and
a plurality of GtG interconnects wherein each GtG interconnect connects two of the GPUs in the first CAC;
wherein each of the node switch interconnects is implemented in accordance with a peripheral components interconnect express (PCIe) interconnect technology; and
wherein each of the one or more IAT interconnects, each of the one or more GtI interconnects, and each of the plurality of GtG interconnects are implemented in accordance with a second interconnect technology wherein the second interconnect technology comprises a greater-than-PCIe (GTP) interconnect technology having an achievable data rate exceeding an achievable data rate of a PCIe interconnect technology.
2. The information handling system of claim 1 , further comprising:
a first inter-node adapter configured to couple the first compute node to an inter-node transport (INT), wherein:
the INT is configured to connect the first inter-node adapter to a second inter-node adapter;
the second inter-node adapter comprises an inter-node adapter of the second compute node; and
the IAT is configured to carry inter-node non-GtG traffic.
3. The information handling system of claim 2 , wherein the node switch interconnects include an interconnect coupling the node switch and the first inter-node adapter.
4. The information handling system of claim 2 , wherein the second interconnect technology includes an upstream sublink, comprising a first plurality of eight or more differential signals, and a downstream sublink, comprising a second plurality of eight or more differential signals.
5. The information handling system of claim 4 , each of the first plurality of eight or more differential signals corresponds to a twisted copper pair.
6. The information handling system of claim 4 , one or more StG interconnects include an StG interconnect corresponding to each of the plurality of GPUs.
7. The information handling system of claim 4 , wherein:
each of the node switch interconnects includes a PCIe link comprising at least 16 bidirectional lanes.
8. The information handling system of claim 4 , wherein the first and second compute nodes are housed in separate chassis and wherein the IAT interface and the INT comprise external interconnects.
9. The information handling system of claim 4 , wherein each of the eight or more differential signals supports 20 giga-transactions per second.
10. The information handling system of claim 4 , wherein each of the GPUs includes:
a switch port configured to receive one of the StG interconnects; and
a plurality of G-ports, each configured to receive a GtG interconnect.
11. The information handling system of claim 4 , wherein:
at least two GtG interconnects connect a pair of the GPUs wherein the pair of the GPUs includes a first GPU and a second GPU; and
at least two GtI interconnects are connected to at least one of the GPUs.
12. The information handling system of claim 4 , wherein:
the first IAT interface includes a plurality of connector blocks; and
the IAT includes a plurality of IAT interconnects, including an IAT interconnect corresponding to each of the plurality of connector blocks.
13. The information handling system of claim 12 , wherein the first inter-node adapter includes an inter-node switch configured to receive an adapter local interconnect connected to the node switch.
14. The information handling system of claim 12 , wherein the plurality of connector blocks includes a plurality of quad small form-factor plugabble double density (QSFP-DD) connectors, each configured to connect one of the GtI interconnects to one of the IAT interconnects.
15. A compute node assembly for use in an information handling system, wherein the compute node assembly comprises:
a central processing unit (CPU);
a first computational accelerator (CAC) comprising a plurality of graphics processing units (GPUs);
a node switch;
a first inter-accelerator transport (TAT) interface configured to receive an IAT comprising one or more IAT interconnects, wherein each IAT interconnect is configured to carry traffic between a GPU in the first CAC and a GPU in a second CAC, wherein the second CAC comprises a CAC of a second compute node; and
compute node interconnects, wherein the compute node interconnects include:
node switch interconnects, wherein the node switch interconnects include:
an interconnect coupling the CPU and the node switch; and
one or more switch-to-GPU (StG) interconnects coupling the node switch and one or more of the plurality of GPUs; and
a plurality of CAC interconnects, including:
one or more (GtI) interconnects, wherein each of the one or more GtI interconnects connects a GPU in the first CAC to the IAT interface; and
a plurality of GPU-to-GPU (GtG) interconnects wherein each GtG interconnect connects two of the GPUs in the first CAC;
wherein each of the node switch interconnects is implemented in accordance with a first interconnect technology; and
wherein each of the one or more IAT interconnects, each of the one or more GtI interconnects, and each of the plurality of GtG interconnects are implemented in accordance with a second interconnect technology wherein the second interconnect technology has an achievable data rate exceeding an achievable data rate of the first interconnect technology.
16. The compute node assembly of claim 15 , wherein the first interconnect technology comprises a peripheral component interconnect express (PCIe) standard and the CAC interconnects comply with a second interconnect standard that differs from the PCIe standard, wherein a maximum data rate of the second interconnect standard exceeds a maximum date rate of the PCIe standard.
17. The compute node assembly of claim 15 , wherein each of the GPUs includes:
a switch port configured to receive a node switch interconnect; and
a plurality of G-ports, each configured to receive a CAC interconnect, wherein the plurality of G-ports include:
one or more G-ports connected to an GtI interconnect; and
one or more G-ports connected to a GtG interconnect.
18. The compute node assembly of claim 17 , wherein the CAC interconnects include:
at least one GtG interconnect for each unique pair of the GPUs; and
at least one GtI interconnect for each of the GPUs.
19. The compute node assembly of claim 18 , wherein:
the first CAC includes four GPUs;
each of the four GPUs includes six G-ports; and
the CAC interconnects include:
six GtI interconnects; and
nine GtG interconnects.
20. The compute node assembly of claim 19 , wherein the first IAT interface includes six external connector blocks, the inter-accelerator transport includes six external cables wherein each external cable of the inter-accelerator transport is connected to one of the connector blocks.