Multi-root sharing of single-root input/output virtualization
In a first embodiment of the present invention, a method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints is provided, the method comprising: CSR redirection to a management processor which either acts as a proxy to execute the CSR request on behalf of the host or filters it and performs an alternate action, downstream routing of memory mapped I/O request packets through the switch in the host's address space and address translation with VF BAR granularity, upstream routing of requests originated by I/O devices by table lookup indexed by Requester ID, and requester ID translation using a fixed local-global RID offset.
1. A method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to the first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints;
executing the CSR request on a virtual function or physical function register, identified in the CSR request, in the first of the plurality of SR-IOV endpoints; and
spooling the CSR request to a buffer allocated to a first root;
wherein the spooling comprises: writing an entire transaction layer packet (TLP) of the CSR request to an address defined in a pointer register;
creating a posted write header using the address defined in the pointer register;
appending the TLP to the posted write header as a payload; and
forwarding a complete packet comprising the posted write header and the appended CSR request TLP to a downstream port identified by the address in the request.
2. The method of claim 1 , further comprising:
locating registers of a virtual physical function for the first of the plurality of SR-IOV endpoints in memory of a centralized management processor;
locating registers of any virtual functions of that shared endpoint that are assigned to a host which originated the CSR request packet and are addressed by it that are located in a memory of the management processor; and
executing the CSR request packet on the appropriate virtual physical function register or virtual function register in the memory of the management processor.
3. The method of claim 1 , further comprising:
translating requester identifiers (IDs) of requests sent from the first host to include the global space bus number of the non-transparent bridge, which provides for completion packets to be returned to the host; and
reverse translating requester IDs of return completions sent to the non-transparent port so that the translated requester ID matches that in the original CSR request packet.
4. The method of claim 2 , further comprising:
receiving a CSR request packet from a second host, wherein the CSR request packet is directed to the first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet from the second host to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints; and
executing the CSR request packet from the second host on an appropriate virtual physical function or virtual function register in the first of the plurality of SR-IOV endpoints, and/or a virtual image or copy of the register in the memory of the management processor.
5. The method of claim 1 , further comprising:
routing transactions sent upstream by the plurality SR-IOR endpoints by global space requester identifier.
6. A method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to a first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints; and
accessing a mapping between host domain addresses of virtual function base addresses and their global domain addressed in order to translate the address in a memory mapped I/O request packet that addresses a register of a virtual function in a first SR-IOV endpoint wherein a translation is provided for each VF Base Address Register (BAR) of each contiguous group of VFs in each SR-IOV endpoint assigned to the first host.
7. A method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to a first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints; and
further comprising:
receiving a memory space request packet addressed to a virtual function of an SR-IOV endpoint; and
forwarding the memory space request packet directly to an appropriate downstream port.
8. The method of claim 7 , wherein the forwarding includes performing address and requester ID translation on the memory space request packet.
9. The method of claim 8 , wherein the translated requester ID of the memory request includes a global space bus number of a non-transparent bridge located at a switch port to which the first host is attached.
10. A system comprising:
a switch having:
a plurality of upstream ports, each upstream port connected to a different host and each represented by both a virtual transparent PCI-PCI bridge and a non-transparent bridge;
a plurality of downstream ports, each downstream port connected to a different SR-IOV endpoint and each represented by a virtual PCI-PCI bridge;
a management processor;
a memory associated with the management processor, wherein the memory contains a plurality of CSR registers that control the virtual PCI-to-virtual PCI bridges;
wherein the switch is configured to:
receive a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to a first of the plurality of SR-IOV endpoints; and
redirect the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints.
11. The system of claim 10 , wherein the management processor is configured to:
locate registers of a virtual physical function for the first of the plurality of SR-IOV endpoints in memory of a centralized management processor;
locate registers of any virtual functions of that shared endpoint that are assigned to a host which originated the CSR request packet and are addressed by it that are located in a memory of the management processor; and
execute the CSR request packet on the appropriate virtual physical function register or virtual function register in the memory of the management processor.
12. The system of claim 11 , wherein the switch is further configured to:
spool the CSR request to a buffer allocated to a first root.
13. The system of claim 12 , wherein the spooling comprises:
writing an entire transaction layer packet (TLP) of the CSR request to an address defined in a pointer register;
creating a posted write header using the address defined in the pointer register;
appending the TLP to the posted write header as a payload; and
forwarding a complete packet comprising the post write header and the appended CSR request TLP to a downstream port identified by the address in the request.
14. The system of claim 10 , wherein the management processor is further configured to:
access a mapping between host domain addresses of virtual function base addresses and global domain addresses in order to translate an address in a memory mapped I/O request packet that addresses a register of a virtual function in a first SR-IOV endpoint wherein a translation is provided for each VF Base Address Register (BAR) of each contiguous group of VFs in each SR-IOV endpoint assigned to the first host.
15. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform a method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to the first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints;
executing the CSR request on a virtual function or physical function register, identified in the CSR request, in the first of the plurality of SR-IOV endpoints; and
spooling the CSR request to a buffer allocated to a first root;
wherein the spooling comprises: writing an entire transaction layer packet (TLP) of the CSR request to an address defined in a pointer register;
creating a posted write header using the address defined in the pointer register;
appending the TLP to the posted write header as a payload; and
forwarding a complete packet comprising the posted write header and the appended CSR request TLP to a downstream port identified by the address in the request.
16. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform a method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to a first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints; and
accessing a mapping between host domain addresses of virtual function base addresses and their global domain addressed in order to translate the address in a memory mapped I/O request packet that addresses a register of a virtual function in a first SR-IOV endpoint wherein a translation is provided for each VF Base Address Register (BAR) of each contiguous group of VFs in each SR-IOV endpoint assigned to the first host.
17. A program storage device readable by a machine tangibly embodying a program of instructions executable by the machine to perform a method for multi-root sharing of a plurality of single root input/output virtualization (SR-IOV) endpoints, the method comprising:
receiving a control and status register (CSR) request packet from a first host, wherein the CSR request packet is a CSR Space packet or a memory mapped I/O request packet, wherein the CSR request packet is addressed to a first of the plurality of SR-IOV endpoints;
redirecting the CSR request packet to a centralized management processor so that the centralized management processor may act as a proxy to communicate with the first of the plurality of SR-IOV endpoints; and
further comprising:
receiving a memory space request packet addressed to a virtual function of an SR-IOV endpoint; and
forwarding the memory space request packet directly to an appropriate downstream port.