Computing system, PCI device manager and initialization method thereof
The present disclosure relates to a computing system, a PCI device manager and an initialization method thereof. The PCI device manager is used to realize peer-to-peer data transmission between PCI devices, and the PCI device manager includes: a DMA engine module that realizes data and command communication with a host and a PCI device through DMA; a device memory; a memory mapping module that maps a data cache space of the device memory to a physical address space of the host memory, so as to perform data communication and/or command communication with the PCI device via the DMA engine module in a running process; and a processor that sends a PCI device configuration instruction to the host through the DMA engine module, and creates PCI device objects respectively for PCI devices discovered by the host.
1 . A peripheral component interconnect (PCI) device manager, used to realize peer-to-peer data transmission between PCI devices, the PCI device manager comprising:
a direct memory access (DMA) engine module connected to a host bus to realize data and command communication with a host and the PCI devices through DMA;
a device memory;
a memory mapping module that maps a data cache space of the device memory to a physical address space of a host memory, so as to perform data communication and/or command communication with the PCI devices via the DMA engine module in a running process; and
a processor that sends a PCI device configuration instruction to the host through the DMA engine module, and creates PCI device objects respectively for PCI devices discovered by the host, so as to realize data communication and/or command communication with the PCI devices discovered by the host through the PCI device objects in the running process.
2 . The PCI device manager according to claim 1 , wherein creating the PCI device objects respectively for the discovered PCI devices comprises:
configuring data cache spaces respectively for the discovered PCI devices in the device memory; and
registering queues respectively for the discovered PCI devices in the device memory.
3 . The PCI device manager according to claim 2 , wherein for a PCI device requesting access to data,
the queues comprise a work queue, wherein the processor reads from the work queue a data access command written by the PCI device corresponding to the work queue; and/or
the queues comprise a completion queue, wherein the processor informs the PCI device that the command execution is completed by writing a completion queue element into the completion queue registered for the PCI device in the device memory.
4 . The PCI device manager according to claim 3 , wherein the data access command specifies a data cache address to be used in this data access operation in the device memory.
5 . The PCI device manager according to claim 2 , wherein for a PCI device storing data,
the queues comprise a work queue, wherein the processor writes a data access command into the work queue registered for the PCI device in the device memory; and/or
the queues comprise a work queue, wherein the processor informs the PCI device to read the data access command from the work queue registered for the PCI device in the device memory by writing a notification message into a doorbell register of the PCI device; and/or
the queues comprise a completion queue, wherein the PCI device informs the PCI device manager that the command execution is completed by writing a completion queue element into the completion queue registered for the PCI device in the device memory.
6 . The PCI device manager according to claim 5 , wherein the data access command specifies a data cache address to be used in this data access operation in the device memory.
7 . The PCI device manager according to claim 2 , wherein when a first PCI device requests to read data stored on a second PCI device,
a first reading command for the second PCI device is written by the first PCI device into a first work queue registered for the first PCI device in the device memory, and the first reading command comprises information of data requested to be read and a specified data cache address in the device memory;
the first reading command is converted into a second reading command corresponding to the second PCI device by the processor, and the second reading command comprises the information of the data requested to be read and the specified data cache address in the device memory;
the second reading command is written into a second work queue registered for the second PCI device in the device memory;
a notification message is written into a doorbell register of the second PCI device through the DMA engine module to inform the second PCI device to read the second reading command from the second work queue;
the data requested to be read is written into a specified data cache area corresponding to the specified data cache address by the second PCI device in response to the notification message; and
in response to completion of a data sending operation of the second PCI device, a notification message is written into a doorbell register of the first PCI device through the DMA engine module to inform the first PCI device to read the data from the specified data cache area.
8 . The PCI device manager according to claim 6 , wherein the first PCI device is a network interface controller and the second PCI device is a solid state disk.
9 . The PCI device manager according to claim 2 , wherein when a first PCI device requests to write data into a second PCI device,
a first writing command for the second PCI device is written by the first PCI device into a first work queue registered for the first PCI device in the device memory, and the first writing command comprises a specified data cache address in the device memory;
data is written into the specified data cache address in the device memory by the first PCI device;
the first writing command is converted into a second writing command corresponding to the second PCI device by the processor, and the second writing command comprises the specified data cache address in the device memory;
the second writing command is written into a second work queue registered for the second PCI device in the device memory; and
a notification message is written into a doorbell register of the second PCI device by the DMA engine module to inform the second PCI device to read the second writing command from the second work queue.
10 . The PCI device manager according to claim 9 , wherein the first PCI device is a network interface controller and the second PCI device is a solid state disk.
11 . A computing system, comprising:
a host, wherein the host comprises a host memory, a host central processing unit (CPU) and a host bus;
multiple peripheral component interconnect (PCI) devices, connected to the host bus; and
a PCI device manager, connected to the host bus and used to realize peer-to-peer data transmission between the multiple PCI devices, comprising: a direct memory access (DMA) engine module connected to the host bus to realize data and command communication with the host and the multiple PCI devices through DMA; a device memory; a memory mapping module that maps a data cache space of the device memory to a physical address space of the host memory, so as to perform data communication and/or command communication with the multiple PCI devices via the DMA engine module in a running process; and a processor that sends a PCI device configuration instruction to the host through the DMA engine module and creates PCI device objects respectively for PCI devices discovered by the host, so as to realize data communication and/or command communication with the PCI devices through the PCI device objects in the running process;
wherein data and command communication between the multiple PCI devices is implemented through the PCI device manager.
12 . An initialization method of the peripheral component interconnect (PCI) device manager of the computing system according to claim 11 , which is suitable for being performed on the host central processing unit (CPU) of the computing system, wherein the method comprises:
registering the device memory in the PCI device manager into a physical address space of the host memory through a PCI base address register space of the host;
receiving a PCI device configuration instruction sent by a program running on the processor of the PCI device manager;
performing a PCI device enumeration instruction to discover PCI devices connected to the host bus;
sending information of the discovered PCI devices to the PCI device manager; and
issuing the PCI device configuration instruction to the discovered PCI devices.
13 . A non-transitory machine-readable storage medium having an executable code stored thereon, wherein the executable code, when executed by a processor of an electronic device, causes the processor to perform the method according to claim 12 .
14 . An initialization method of the peripheral component interconnect (PCI) device manager of the computing system according to claim 11 , which is suitable for being performed on the processor of the PCI device manager of the computing system, wherein the method comprises:
sending a PCI device configuration instruction to the host; and
creating PCI device objects respectively for discovered PCI devices based on information of the PCI devices discovered by the host returned from the host, so as to realize data communication and/or command communication with the PCI devices through the PCI device objects in a running process.
15 . A non-transitory machine-readable storage medium having an executable code stored thereon, wherein the executable code, when executed by a processor of an electronic device, causes the processor to perform the method according to claim 14 .
16 . The computing system according to claim 11 , wherein creating the PCI device objects respectively for the discovered PCI devices comprises:
configuring data cache spaces respectively for the discovered PCI devices in the device memory; and
registering queues respectively for the discovered PCI devices in the device memory.
17 . The computing system according to claim 16 , wherein for a PCI device requesting access to data,
the queues comprise a work queue, wherein the processor reads from the work queue a data access command written by the PCI device corresponding to the work queue; and/or
the queues comprise a completion queue, wherein the processor informs the PCI device that the command execution is completed by writing a completion queue element into the completion queue registered for the PCI device in the device memory.
18 . The computing system according to claim 16 , wherein for a PCI device storing data,
the queues comprise a work queue, wherein the processor writes a data access command into the work queue registered for the PCI device in the device memory; and/or
the queues comprise a work queue, wherein the processor informs the PCI device to read the data access command from the work queue registered for the PCI device in the device memory by writing a notification message into a doorbell register of the PCI device; and/or
the queues comprise a completion queue, wherein the PCI device informs the PCI device manager that the command execution is completed by writing a completion queue element into the completion queue registered for the PCI device in the device memory.
19 . The computing system according to claim 16 , wherein when a first PCI device requests to read data stored on a second PCI device,
a first reading command for the second PCI device is written by the first PCI device into a first work queue registered for the first PCI device in the device memory, and the first reading command comprises information of data requested to be read and a specified data cache address in the device memory;
the first reading command is converted into a second reading command corresponding to the second PCI device by the processor, and the second reading command comprises the information of the data requested to be read and the specified data cache address in the device memory;
the second reading command is written into a second work queue registered for the second PCI device in the device memory;
a notification message is written into a doorbell register of the second PCI device through the DMA engine module to inform the second PCI device to read the second reading command from the second work queue;
the data requested to be read is written into a specified data cache area corresponding to the specified data cache address by the second PCI device in response to the notification message; and
in response to completion of a data sending operation of the second PCI device, a notification message is written into a doorbell register of the first PCI device through the DMA engine module to inform the first PCI device to read the data from the specified data cache area.
20 . The computing system according to claim 16 , wherein when a first PCI device requests to write data into a second PCI device,
a first writing command for the second PCI device is written by the first PCI device into a first work queue registered for the first PCI device in the device memory, and the first writing command comprises a specified data cache address in the device memory;
data is written into the specified data cache address in the device memory by the first PCI device;
the first writing command is converted into a second writing command corresponding to the second PCI device by the processor, and the second writing command comprises the specified data cache address in the device memory;
the second writing command is written into a second work queue registered for the second PCI device in the device memory; and
a notification message is written into a doorbell register of the second PCI device by the DMA engine module to inform the second PCI device to read the second writing command from the second work queue.