Apparatus and methods for clustering multiple independent PCI express hierarchies
View Patent ↗Apparatus, systems and methods for clustering multiple PCI Express hierarchies to enable access of components in different hierarchies. Each PCI Express hierarchy includes a root device as well as a cluster port for coupling each hierarchy to each other hierarchy of a physical cluster through a PCI Express switched fabric. Memory addresses and transaction IDs are re-mapped by the cluster port of each hierarchy to partition the PCI Express space of the system of multiple hierarchies. A first portion of the partitioned space is reserved for access to local components within a hierarchy. A second portion of the space is used to access remote components of other hierarchies from within a different first hierarchy. The address and transaction ID values exchanged in such remote transactions are re-mapped and used by the cluster port of each hierarchy to route exchanges between hierarchies using standard PCIe root devices, endpoint devices, and switches.
1. A system comprising:
a plurality of PCI Express controllers each comprising a PCI Express hierarchy, each hierarchy having an associated root controller, and each hierarchy having an associated cluster port; and
a PCI Express switch coupled to the cluster port of each hierarchy through a physical PCI Express bus, wherein the PCI Express switch is not multi-root aware (“MRA”),
wherein the cluster port of each hierarchy enables controlled access through the switch by a first hierarchy to components within any other hierarchy of the plurality of hierarchies, and
wherein the cluster port of each hierarchy partitions the PCI Express space of the system such that a first portion of the space is used within said each hierarchy to reference local components within said each hierarchy and such that a second portion of the space is used within said each hierarchy to reference remote components within other hierarchies of the plurality of hierarchies.
2. The system of claim 1
wherein the cluster port of each hierarchy further comprises:
a memory address mapping element for mapping memory addresses of the PCI Express space, the mapping element operable to translate memory addresses between addresses associated with local components of the first hierarchy containing the cluster port and addresses of remote components of another hierarchy.
3. The system of claim 2 , wherein the memory address mapping element is a memory address translator,
wherein the memory address translator is adapted to associate each hierarchy other than the first hierarchy with a segment of the PCI Express space having a non-zero value in designated high order address bits of a memory address,
wherein the memory address translator is further adapted to associate local components of the first hierarchy with a segment of the PCI Express space having a zero value in the designated high order address bits, and
wherein the designated high order address bits comprises a plurality of contiguous bits, the plurality of contiguous bits comprising a most significant bit but not a least significant bit.
4. The system of claim 1
wherein the cluster port of each hierarchy further comprises:
a configuration space translator for translating a PCI Express transaction ID in the PCI Express space, the translator operable to translate the transaction ID exchanged between the first hierarchy containing the cluster port and another hierarchy.
5. The system of claim 4
wherein the system is adapted to reserve a plurality of bits in one or more fields of the transaction ID, and
wherein the configuration space translator is adapted to save an original bus number of the transaction ID in the reserved bits.
6. The system of claim 5
wherein the system is adapted to reserve five bits from the one or more fields of the transaction ID.
7. The system of claim 5
wherein the system is adapted to reserve 3 bits in the bus number field of the transaction ID and 2 bits in the device number field of the transaction ID and 3 bits in the tag field of the transaction ID.
8. A cluster port in a PCI Express controller comprising a PCI Express hierarchy, the cluster port comprising:
a local interface to a PCI Express switch for coupling the cluster port to a local hierarchy;
a remote interface for coupling, through a physical PCI Express bus, the cluster port to remote components in one or more other PCI Express hierarchies;
a configuration store for storing PCI Express configuration information relating to the local hierarchy and relating to the other hierarchies; and
a translator element for translating PCI Express transactions between the local hierarchy and any of the other hierarchies, the translator element comprising a memory address mapping element for mapping memory addresses of the PCI Express space, the mapping element operable to translate memory addresses from addresses associated with local components of the local hierarchy to addresses associated with remote components of the other hierarchies,
wherein the cluster port enables controlled access between component of the local hierarchy and remote components within the one or more other hierarchies, and
wherein the cluster port is adapted to partition the PCI Express space of the system such that a first portion of the space is used within the local hierarchy to reference local components within local hierarchy and such that a second portion of the space is used to reference remote components within another of the other hierarchies.
9. The cluster port of claim 8 , wherein the memory address mapping element is a memory address translator,
wherein the memory address translator is adapted to associate each other hierarchy with a segment of the PCI Express space having a non-zero value in designated high order address bits of a memory address,
wherein the memory address translator is further adapted to associate local components of the local hierarchy with a segment of the PCI Express space having a zero value in the designated high order address bits and
wherein the designated high order address bits comprises a plurality of contiguous bits, the plurality of contiguous bits comprising a most significant bit but not a least significant bit.
10. The cluster port of claim 8
wherein the translator element further comprises:
a configuration space translator for translating a PCI Express transaction ID in the PCI Express space, the translator operable to translate the transaction ID exchanged between the local hierarchy and another hierarchy.
11. The cluster port of claim 10
wherein the cluster port is adapted to reserve a first plurality of bits in one or more fields of the transaction ID; and
wherein the cluster port is further adapted to use only a second plurality of bits in one field of the transaction ID to fully identify the local hierarchy by a particular hierarchy of the other hierarchies.
12. The cluster port of claim 11
wherein the cluster port is adapted to reserve eight bits from the one or more fields of the transaction ID.
13. The cluster port of claim 12
wherein the cluster port is adapted to reserve 3 bits in the bus number field of the transaction ID and 2 bits in the device number field of the transaction ID and 3 bits in the tag field of the transaction ID.
14. The cluster port of claim 8 , wherein the remote interface couples with the remote components through another PCI Express switch.
15. A method comprising:
providing a plurality of PCI Express controllers, each controller comprising a hierarchy having an associated root controller and each hierarchy having an associated cluster port;
designating one hierarchy of the plurality of hierarchies as a supervisory hierarchy,
providing a PCI Express switch coupled to the cluster port of each hierarchy, wherein the PCI Express switch is not multi-root aware (“MRA”);
configuring the cluster port of each hierarchy to permit access to selected components of said each hierarchy by other components of other hierarchies; and
accessing a component of another hierarchy from a first hierarchy through its corresponding cluster port.
16. The method of claim 15
wherein the step of configuring further comprises:
partitioning the PCI Express space of each hierarchy of the plurality of hierarchies such that a first portion of the space is used within said each hierarchy to reference local components within said each hierarchy and such that a second portion of the space is used within said each hierarchy to reference remote components within other hierarchies of the plurality of hierarchies.
17. The method of claim 16
wherein the step of accessing further comprises:
operating the cluster port to map memory addresses of the PCI Express space by translating memory addresses between addresses associated with local components of the first hierarchy containing the cluster port and addresses of remote components of another hierarchy.
18. The method of claim 16
wherein the step of accessing further comprises:
operating the cluster port to translate a PCI Express transaction ID exchanged between the first hierarchy containing the cluster port and another hierarchy.
19. The method of claim 16
wherein the step of configuring further comprises:
initializing each of the plurality of hierarchies to access components within its local hierarchy;
initializing the cluster port of said each hierarchy in accordance with the first and second portions defined by the step of partitioning; and
initializing the PCI Express switch in accordance with the first and second portions defined by the step of partitioning.
20. The method of claim 19
wherein the step of initializing the PCI Express switch is performed by operation of the supervisory hierarchy.