System and methods of managing and recognizing PCI devices in an active system
In part, in one aspect, the disclosure relates to a method of enumerating a device relative to a computer system. The method may include sending, using a platform driver of an operating system (OS), a PCI memory range to self-enumeration (SE) firmware once it detects a new PCIe module; establishing a communication channel between the platform driver and SE firmware; detecting, using the platform driver, the new PCIe module via the PCI hotplug capability of the OS; and configuring a communication device, using the platform driver, on the PCIe module to establish a communication channel to the SE firmware. In some embodiments, the communication device is a synthetic device on the PCIe switch that allows low bandwidth bi-directional communication.
1 . A method of recognizing Peripheral Component Interconnect (PCI) devices in an active system, the method comprising:
inserting a Peripheral Component Interconnect Express (PCIe) module in a first computer system running a first operating system, wherein the first operating system lacks support for hot plug insertion of a PCI device,
establishing a first communication channel between the first operating system and the PCIe module;
transmitting a memory range from the first operating system to the PCIe module, the memory range comprising a range of memory addresses of a first PCI bridge, wherein the PCIe module is in communication with the first PCI bridge;
setting up a PCI hierarchy for the PCIe module, wherein the PCI hierarchy comprises the first PCI bridge, one or more PCI bridges, and one or more PCI devices, wherein the one or more PCI devices are each connected to at least one of the first PCI bridge and the one or more PCI bridges;
resetting connection between the first operating system and the PCIe module; and
establishing a second communication channel between the first operating system and the PCIe module, the first operating system recognizing the PCI hierarchy in response to establishing a link to the PCIe module.
2 . The method of claim 1 , wherein the first operating system comprises a platform driver, wherein the platform driver is a kernel mode driver.
3 . The method of claim 2 , wherein the PCIe module comprises a switch, wherein the switch comprises switch firmware.
4 . The method of claim 3 , wherein the first communication channel is established between the switch firmware and the platform driver.
5 . The method of claim 3 , wherein the platform driver transmits the memory range to the switch firmware.
6 . The method of claim 5 , wherein the switch firmware sets up the PCI hierarchy.
7 . The method of claim 3 further comprising booting the switch firmware.
8 . The method of claim 7 further comprising detecting which PCI bus of the first computer is connected to the PCIe module.
9 . The method of claim 7 further comprising configuring one or more PCI bus registers of a communication device of the switch to allow the first operating system to detect the communication device.
10 . The method of claim 9 , wherein the platform driver recognizes the communication device, wherein the first communication channel is established between the platform driver and the switch firmware using the communication device.
11 . The method of claim 3 , wherein the first PCI bridge is a root bridge to which the switch is attached.
12 . The method of claim 11 , wherein transmitting the memory range is performed by the platform driver, wherein the memory range is transmitted to the switch firmware.
13 . The method of claim 12 further comprising specifying a PCI configuration space of all bridges in the PCI hierarchy.
14 . The method of claim 12 further comprising determining memory ranges for each PCI bridge on or connected to the switch using the memory range.
15 . The method of claim 12 further comprising booting the switch firmware; obtaining a set of PCI bus numbers during boot; and determining a set of bus numbers of PCI bridges on the switch using the set of PCI bus numbers.
16 . The method of claim 15 , wherein setting up a PCI hierarchy for the PCIe module further comprises writing values into PCI bus registers and PCI memory registers of the bridge, wherein the values comprise the memory ranges for each PCI bridge and the PCI bus numbers, respectively.
17 . The method of claim 15 further comprising communicating, from the switch firmware to the platform driver, that completion of self-enumeration of the PCI hierarchy is complete.
18 . The method of claim 3 , wherein step of resetting connection between the first operating system and the PCIe module further comprises the platform driver terminating link to the switch and then establishing the link to the switch.
19 . The method of claim 3 further comprising bringing the PCI hierarchy and the PCI devices in the PCI hierarchy online using the first operating system.
20 . The method of claim 1 , wherein the first operating system is Elastic Sky X (ESX).
21 . The method of claim 1 , wherein one or more steps of the method are implemented on one or more compute nodes that comprise a PCI bridge on the PCIe module that is connected to a root port of the root bridge on the host system.
22 . The method of claim 21 , wherein PCI devices on the PCIe module are connected to other PCI bridges on the PCIe module.
23 . A method of enumerating a device relative to a computer system, the method comprising:
inserting a new Peripheral Component Interconnect Express (PCIe) module in a first computer system running a first operating system (OS), wherein the OS requires preconfiguring of a PCIe device to support hot plug insertion of the PCIe device;
sending, using a platform driver of the OS, a PCI memory range to self-enumeration (SE) firmware once it detects the new PCIe module, wherein the platform driver is a kernel mode driver;
establishing a communication channel between the platform driver and SE firmware, wherein the PCIe module comprises the self-enumeration (SE) firmware;
detecting, using the platform driver, the new PCIe module; and
configuring a communication device, using the platform driver, on the PCIe module to establish a communication channel to the SE firmware.
24 . The method of claim 23 , wherein communication device is a device that allows bi-directional communication.
25 . The method of claim 24 , wherein the PCIe module comprises a PCIe switch, wherein the PCIe switch further comprises the SE firmware.