Unified system area network and switch
A network switch, based on the PCI Express protocol, is disclosed. The switch includes a processor, local memory and a plurality of non-transparent bridges. By configuring the non-transparent bridges appropriately, the network switch can facilitate a number of different communication mechanisms, including TCP/IP communication between servers, server clusters, and virtualized I/O device utilization. For example, the network switch may configure the non-transparent bridges so as to have access to the physical memory of every server attached to it. It can then move data from the memory of any server to the memory of any other server. In another embodiment, the network switch is connected to an I/O device, and multiple servers are given access to that I/O device via virtualized connections.
1. A method of clustering two servers, each server having a local memory and processor, comprising:
connecting each of said two servers to a switch using PCI Express, said switch comprising a processor, a local memory and a plurality of non-transparent bridges, a first of said non-transparent bridges used to connect to a first server and a second of said non-transparent bridges used to connect to a second server;
configuring said first non-transparent bridge so that a first portion of said local memory of said first server is mapped into a logical address space of said processor in said switch; thereby allowing said switch to access said first portion of said local memory of said first server;
configuring said second non-transparent bridge so that a second portion of said local memory of said second server is mapped into said logical address space of said processor in said switch; thereby allowing to allow said switch to access said second portion of local memory of said second server;
using said first server to write data destined for said second server to said first portion;
using said network switch to copy data from said first portion using said first non-transparent bridge to said second portion using said second non-transparent bridge; and
using said second server to read data from said second portion.
2. The method of claim 1 , wherein a direct memory access (DMA) engine located in said network switch is used to read said data from said first portion and copy said data to said second portion.
3. The method of claim 1 , wherein said first non-transparent bridge comprises a messaging register to allow communication between said processor of said first server and a processor in said network switch, and said processor of said first server notifies said network switch of the existence of said data in said first portion using said messaging register.
4. The method of claim 1 , wherein said second non-transparent bridge comprises a messaging register to allow communication between said processor of said second server and a processor in said network switch, and said network switch notifies said second server of the existence of said data in said second portion using said messaging register.
5. The method of claim 1 , wherein said first server, said second server and said network switch are attached via a network connection and said processor of said first server notifies said network switch of the existence of said data in said first portion using said network connection.
6. The method of claim 1 , wherein said first server, said second server and said network switch are attached via a network connection and said processor of said network switch notifies said second server of the existence of said data in said second portion using said network connection.
7. A system for clustering two servers, comprising:
a first server having a first local memory, a first processor and a first PCI Express port,
A second server having a second local memory, a second processor and a second PCI Express port;
network switch comprising a third processor, a third local memory, a first non-transparent bridge connected to said first PCI Express port, and a second non-transparent bridge connected to said second PCI express port, wherein said third local memory further comprises computer executable instructions adapted to:
configure said first non-transparent bridge so that a first portion of said local memory of said first server is mapped into a logical address space of said processor in said switch; thereby allowing said network switch to access said first portion of said first local memory;
configure said second non-transparent bridge so that a second portion of said local memory of said second server is mapped into a logical address space of said processor in said switch; thereby allowing said network switch to access said second portion of said second local memory;
receive a notification from said first server of new data written in said first portion;
copy data from said first portion to said second portion using said first and second non-transparent bridges, in response to said notification; and
notify said second server that new data is available in said second portion after copying said data to said second portion.
8. The system of claim 7 , further comprising a DMA engine in said network switch to copy said data from said first portion to said second portion.
9. The system of claim 7 , wherein said non-transparent bridges each comprise a messaging register, and said notifications are transmitted using said messaging register.
10. The system of claim 7 , wherein said first server, said second server and said network switch are attached via a network connection and said notifications are transmitted using said network connection.
11. A method of clustering two servers, each server having a local memory and processor, comprising:
connecting each of said two servers to a switch using PCI Express, said switch comprising a plurality of non-transparent bridges, a first of said non-transparent bridges used to connect to a first server and a second of said non-transparent bridges used to connect to a second server;
configuring said first non-transparent bridge such that accesses from said first server and accesses from said second server utilize the same logical address range in said switch, such that read and write operations from said first server via said first non-transparent bridge are claimed by said second non-transparent bridge and read and write operations from said second server via said second non-transparent bridge are claimed by said first non-transparent bridge;
using said first server to perform read and write operations, whereby said operations are performed on said local memory of said second server; and
using said second server to perform read and write operations, whereby said operations are performed on said local memory of said first server.
12. A system for clustering two servers, comprising:
a first server having a first local memory, a first processor and a first PCI Express port,
A second server having a second local memory, a second processor and a second PCI Express port;
network switch comprising a third processor, a third local memory, a first non-transparent bridge connected to said first PCI Express port, and a second non-transparent bridge connected to said second PCI express port, wherein said third local memory further comprises computer executable instructions configured to:
configure said first non-transparent bridge such that accesses from said first server and accesses from said second server utilize the same logical address range in said switch, such that read and write operations from said first server via said first non-transparent bridge are claimed by said second non-transparent bridge and read and write operations from said second server via said second non-transparent bridge are claimed by said first non-transparent bridge.