PCI express fabric routing for a fully-connected mesh topology

1. A PCIe Fabric comprising:

an Input/Output (I/O) tier comprising a first IO tier switch and a second IO tier switch;

a hub tier comprising a plurality of hub tier switches, wherein the plurality of hub tier switches comprises a first hub tier switch and a second hub tier switch, wherein the first IO tier switch is connected to the first hub tier switch using a first link, wherein the first IO tier switch is connected to second hub tier switch using a crosslink;

a plurality of target devices, wherein each of the plurality of target devices is connected to one of the plurality of hub tier switches, wherein the first hub tier switch is connected to a first target device of the plurality of target devices using a second link, and wherein the first hub tier switch is connected to a second target device of the plurality of target devices using a third link;

wherein the first IO tier switch is configured to:

receive, from a client, a transaction layer packet (TLP);

make a first determination that an address in the TLP is not associated with any multicast address range in the first IO tier switch;

make a second determination that the address in TLP is not associated with any downstream port in the first IO tier switch;

based on the first and second determinations, route the TLP to the first hub tier switch using the first link via a first upstream port on the first IO tier switch;

wherein the first hub tier switch is configured to:

receive, from the first IO tier switch, the TLP at a first downstream port;

make a third determination that the TLP is associated with a first multicast group;

based on the third determination:

generate a first rewritten TLP and route the first rewritten TLP to the first target device using the second link via a second downstream port on the first hub tier switch, wherein the second downstream port is a subscriber of the first multicast group.

2. The PCIe fabric of claim 1 , wherein the first hub tier switch is further configured to:

generate a second rewritten TLP and route the second rewritten TLP to the first target device using the third link via a third downstream port on the first hub tier switch, wherein the third downstream port is a subscriber of the first multicast group.

3. The PCIe fabric of claim 1 , further comprising:

a processor, wherein the processor is connected to the first hub tier switch using a fourth link, wherein the fourth link is between a second upstream port on the hub tier switch and a root port on the processor,

wherein the root port is associated with the second multicast group,

wherein the first hub tier switch is further configured to:

based on the third determination, generate a third rewritten TLP and route the third rewritten TLP to the processor using the third link via the second upstream port on the first hub tier switch, wherein the second upstream port is a subscriber of the first multicast group.

4. The PCIe fabric of claim 3 , further comprising:

a second processor, wherein the processor and the second processor are operatively connected via a processor interconnect.

5. The PCIe Fabric of claim 1 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a fourth determination that the second TLP comprises an address that is within an address range associated with a third downstream port on the first IO tier switch;

based on the fourth determination, route the second TLP to the second hub tier switch using the crosslink via the third downstream port;

wherein the second hub tier switch is configured to:

receive, from the first IO tier switch, the second TLP at a fourth downstream port;

make a fifth determination that the second TLP is associated with the target device directly connected to the second hub tier switch;

based on the fifth determination, route the second TLP to the target device directly connected to the second hub tier switch.

6. The PCIe Fabric of claim 1 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a fourth determination that the second TLP comprises an address that is within an address range associated with a third downstream port on the first IO tier switch;

based on the fourth determination, route the second TLP to the second hub tier switch using the crosslink via the third downstream port;

wherein the second hub tier switch is configured to:

receive, from the first IO tier switch, the second TLP at a fourth downstream port;

make a fifth determination that the second TLP is associated a second multicast group:

based on the fifth determination, generate a third rewritten TLP and route the third rewritten TLP to a third target device using a fourth link via a fifth downstream port on the second hub tier switch, wherein the fifth downstream port is a subscriber of the second multicast group, and wherein the third target device is connected to the second hub tier switch via the fourth link.

7. The PCIe Fabric of claim 1 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a third determination that the second TLP is associated with the first multicast group;

based on the third determination, generate a second rewritten TLP and route the second rewritten TLP to the first hub tier switch using the first link;

wherein the first hub tier switch is configured to:

receive, from the first IO tier switch, the second rewritten TLP at the first downstream port;

make a fifth determination that the second rewritten TLP is associated with the target device directly connected to the first hub tier switch;

based on the fifth determination, route the second rewritten TLP to the target device directly connected to the second hub tier switch.

8. The PCIe fabric of claim 1 , wherein the first upstream port on the first IO tier switch is associated with a first address range that includes all memory in the PCIe fabric excluding memory directly connected to the processor and a second address range that includes all memory associated with clients connected to the PCIe fabric, wherein the first address range and the second address range are used to route TLP packets received by the first IO tier switch.

9. The PCIe fabric of claim 8 , wherein all the memory associated with the clients comprises at least one selected from a group consisting of persistent storage and volatile memory.

10. The PCIe fabric of claim 8 , wherein all the memory in the PCIe fabric comprises at least one selected from a group consisting of persistent storage and volatile memory.

11. The PCIe fabric of claim 1 , wherein the second upstream port on the first hub tier switch is associated with a first address range that includes all memory in all slices in the PCIe fabric and a second address range that includes all memory associated with clients connected to the PCIe fabric, wherein the first IO tier switch and the first hub tier switch are in a first slice of the slices in the PCIe fabric, wherein the first slice excludes memory directly connected to the processor and excludes any of the plurality of the target devices that are not directly connected to the first hub tier switch, wherein the first address range and the second address range are used to route TLP packets received by the first IO tier switch.

12. The PCIe fabric of claim 1 , wherein the crosslink connects a third downstream port of the first IO tier switch and a downstream port of the second hub tier switch.

13. The PCIe fabric of claim 12 , wherein the third downstream port of the first IO tier is associated with an address range that includes memory in a slice of the PCIe fabric, wherein the memory in the slice of the PCIe fabric includes the plurality of the target devices that are directly connected to the second hub tier switch and excludes any of the plurality of the target devices that are directly connected to the first hub tier switch.

14. The PCIe fabric of claim 1 , wherein at least one of the plurality of target devices comprises at least one selected from a group consisting of volatile memory and persistent storage.

15. The PCIe fabric of claim 1 , wherein at least one of the plurality of target devices comprises NAND flash.

16. The PCIe fabric of claim 1 , wherein at least one of the plurality of target devices is one selected from a group consisting of a network endpoint device and a storage endpoint device.

17. The PCIe fabric of claim 1 , wherein the client is a PCIe endpoint.

18. The PCIe fabric of claim 1 , wherein the first IO tier switch and the first hub tier switch are connected using a second crosslink.

19. The PCIe fabric of claim 18 , wherein the first IO tier switch is configured to route TLPs to the first hub tier switch using the second crosslink.

20. The PCIe fabric of claim 18 , wherein the second crosslink is not associated with any multicast group.

21. The PCIe fabric of claim 18 , wherein the second crosslink is connected to a first downstream port on the first IO tier switch and a third downstream port on the first hub tier switch.

22. A PCIe Fabric comprising:

an Input/Output (I/O) tier comprising a first IO tier switch and a second IO tier switch;

a hub tier comprising a plurality of hub tier switches, wherein the plurality of hub tier switches comprises a first hub tier switch and a second hub tier switch, wherein the first IO tier switch is connected to the first hub tier switch using a first link, wherein the first IO tier switch is connected to second hub tier switch using a crosslink;

wherein the PCIe fabric is operatively connected to a plurality of target devices, wherein each of the plurality of target devices is connected to one of the plurality of hub tier switches, wherein the first hub tier switch is connected to a first target device of the plurality of target devices using a second link, and wherein the first hub tier switch is connected to a second target device of the plurality of target devices using a third link;

wherein the first IO tier switch is configured to:

receive, from a client, a transaction layer packet (TLP);

make a first determination that an address in the TLP is not associated with any multicast address range in the first IO tier switch;

make a second determination that the address in TLP is not associated with any downstream port in the first IO tier switch;

based on the first and second determinations, route the TLP to the first hub tier switch using the first link via a first upstream port on the first IO tier switch;

wherein the first hub tier switch is configured to:

receive, from the first IO tier switch, the TLP at a first downstream port;

make a third determination that the TLP is associated with a first multicast group;

based on the third determination:

generate a first rewritten TLP and route the first rewritten TLP to the first target device using the second link via a second downstream port on the first hub tier switch, wherein the second downstream port is a subscriber of the first multicast group.

23. The PCIe Fabric of claim 22 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a fourth determination that the second TLP comprises an address that is within an address range associated with a third downstream port on the first IO tier switch;

based on the fourth determination, route the second TLP to the second hub tier switch using the crosslink via the third downstream port;

wherein the second hub tier switch is configured to:

receive, from the first IO tier switch, the second TLP at a fourth downstream port;

make a fifth determination that the second TLP is associated with the target device directly connected to the second hub tier switch;

based on the fifth determination, route the second TLP to the target device directly connected to the second hub tier switch.

24. The PCIe Fabric of claim 22 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a fourth determination that the second TLP comprises an address that is within an address range associated with a third downstream port on the first IO tier switch;

based on the fourth determination, route the second TLP to the second hub tier switch using the crosslink via the third downstream port;

wherein the second hub tier switch is configured to:

receive, from the first IO tier switch, the second TLP at a fourth downstream port;

make a fifth determination that the second TLP is associated a second multicast group:

based on the fifth determination, generate a third rewritten TLP and route the third rewritten TLP to a third target device using a fourth link via a fifth downstream port on the second hub tier switch, wherein the fifth downstream port is a subscriber of the second multicast group, and wherein the third target device is connected to the second hub tier switch via the fourth link.

25. The PCIe Fabric of claim 22 ,

wherein the first IO tier switch is further configured to:

receive, from the client, a second TLP;

make a third determination that the second TLP is associated with the first multicast group;

based on the third determination, generate a second rewritten TLP and route the second rewritten TLP to the first hub tier switch using the first link;

wherein the first hub tier switch is configured to:

receive, from the first IO tier switch, the second rewritten TLP at the first downstream port;

make a fifth determination that the second rewritten TLP is associated with the target device directly connected to the first hub tier switch;

based on the fifth determination, route the second rewritten TLP to the target device directly connected to the second hub tier switch.