Methods and apparatus for high-speed data bus connection and fabric management
Methods and apparatus for efficient scaling of fabric architectures such as those based on PCIe technology, including up to very large fabrics and numbers of hosts/devices for use in ultra-high performance applications such as for example data centers and computing clusters. In one aspect, methods and apparatus for using Non-Transparent Bridge (NTB) technology to export Message Signaled Interrupts (MSIs) to external hosts are described. In a further aspect, an IO Virtual Address (IOVA) space is created is used as a method of sharing an address space between hosts, including across the foregoing NTB(s). Additionally, a Fabric Manager (FM) entity is disclosed and utilized for programming e.g., PCIe switch hardware to effect a desired host/fabric configuration.
1. Network apparatus for use in a multi-host device system, the network apparatus configured to perform at least one of base address register (BAR) space hiding or BAR space conservation in order to enable accommodation of a plurality of host devices which would otherwise exceed a BAR space capability of the multi-host device system without either of said space hiding or BAR space conservation, the network apparatus comprising a storage medium with at least one computer program configured to, when executed by a digital processing device, cause the network apparatus to:
allocate data fabric address spaces to base address registers (BARs) of the plurality of host devices, wherein at least a portion of the data fabric address spaces comprises a private space not visible to the plurality of host devices; and
assign unique IDs to the plurality of host devices, wherein at least a portion of the unique IDs are multiplexed to a single private ID via at least utilization of at least a first portion and a second portion of a transaction layer packet (TLP) comprising the single private ID, the first portion of the TLP used to enable the multiplexing.
2. A network architecture for use with a plurality of host devices and a plurality of processors each comprising at least one respective root complex, the network architecture comprising:
non-transparent bridge (NTB) data fabric connected to the plurality of host devices, the NTB data fabric comprising a plurality of edge switches and IO (input/output) switches; and
computerized logic configured to utilize a known topology of the NTB data fabric to statically define a single common virtual address map associated with a shared interrupt memory space, wherein the shared interrupt memory space comprises portions of host device memory that are exposed to each of the plurality of host devices connected to the NTB data fabric;
wherein a first of the plurality of host devices may communicate one or more interrupts with a target one of the plurality of host devices via the NTB data fabric via dynamic software-based allocation of said one or more interrupts, said dynamic software-based allocation being without (i) access of a configuration space associated with the target one of the plurality of host devices, and (ii) use of an enumerated device to perform the dynamic software-based allocation.
3. The network architecture of claim 2 , wherein each of the plurality of processors is configured to define how the memory space is translated via a respective input/output memory management unit (IOMMU).
4. A network architecture for use with a plurality of host devices and a plurality of processors each comprising at least one respective root complex, the network architecture configured to enable symmetric peer-to-peer communication between each of the plurality of host devices, the network architecture comprising:
non-transparent bridge (NTB) data fabric connected to the plurality of host devices, the NTB data fabric comprising a plurality of edge switches and IO (input/output) switches and configured to utilize a PCIe protocol for address-based routing of interrupts without use of doorbell registers; and
computerized logic configured to utilize a known topology of the NTB data fabric to statically define a single common virtual address map associated with a shared memory space, wherein the shared memory space comprises portions of host device memory that are exposed to each of the plurality of host devices connected to the NTB data fabric, thereby enabling said symmetric peer-to-peer communication between at least any two of the plurality of host devices.
5. The network architecture of claim 4 , wherein the symmetric peer-to-peer communication comprises each of (i) response to a request; and (ii) initiation of an unsolicited request.
6. The network architecture of claim 4 , wherein the shared memory space comprises a plurality of ranges corresponding to respective ones of the plurality of host devices, and wherein the plurality of ranges can be assigned dynamically.
7. A method for using non-transparent bridge (NTBs) entities for transmitting interrupt data across a managed data fabric connected to a plurality of physically distributed, enumerated devices in support of networking functionality, the method comprising:
defining a common message signaled interrupt space within the managed data fabric, wherein the plurality of physically distributed, enumerated devices comprises at least a plurality of network ports and wherein the common message signaled interrupt space is shared by at least a portion of the plurality of network ports;
defining at least one virtual message signaled interrupt; and
transmitting at least one instance of the at least one virtual message signaled interrupt from a first network port to at least one second network port of the at least portion of the plurality of network ports;
wherein the defining the at least one virtual message signaled interrupt comprises defining the at least one virtual message signaled interrupt without an enumerated device requesting the at least one virtual message signaled interrupt.