Topology aware real time GPU-to-GPU traffic monitoring method and analyzing tools
An information handling system may include at least one central processing unit (CPU), and a plurality of special-purpose processing units. The information handling system may be configured to: repeatedly monitor amounts of data traffic exchanged between respective pairs of the special-purpose processing units during selected time periods; based on the monitored amounts of data traffic, determine an amount of traffic imbalance for the plurality of special-purpose processing units; and in response to the amount of traffic imbalance exceeding a threshold amount, transmit a notification indicative of the amount of traffic imbalance.
1. An information handling system comprising:
at least one central processing unit (CPU); and
a plurality of special-purpose processing units;
wherein the information handling system is configured to:
execute a driver program for controlling the plurality of special-purpose processing units, wherein the driver program is configured to access a counter repeatedly to monitor amounts of data traffic exchanged between respective pairs of the special-purpose processing units during selected time periods that are determined based on a monitoring policy;
subsequent to each selected time period, cause the counter to be reset;
based on the monitored amounts of data traffic, determine an amount of traffic imbalance for the plurality of special-purpose processing units; and
in response to the amount of traffic imbalance exceeding a threshold amount, transmit a notification indicative of the amount of traffic imbalance.
2. The information handling system of claim 1 , wherein the special-purpose processing units are graphics processing units (GPUs).
3. The information handling system of claim 2 , wherein the GPUs are configured as general-purpose graphics processing units (GPGPUs).
4. The information handling system of claim 1 , wherein the repeatedly monitoring comprises periodically monitoring.
5. The information handling system of claim 1 , wherein the information handling system is further configured to:
determine a most-utilized one of the plurality of special-purpose processing units.
6. The information handling system of claim 1 , wherein the information handling system is further configured to:
provide a real-time graphical display indicative of the amounts of data traffic.
7. The information handling system of claim 6 , wherein the real-time graphical display is further indicative of a topology of the plurality of special-purpose processing units.
8. The information handling system of claim 1 , wherein the at least one CPU is a component of a management controller of the information handling system.
9. A method comprising:
executing a driver program for controlling a plurality of special-purpose processing units of an information handling system, wherein the driver program is configured to access a counter repeatedly to monitor amounts of data traffic exchanged between respective pairs of the plurality of special-purpose processing units during selected time periods that are determined based on a monitoring policy;
subsequent to each selected time period, causing the counter to be reset;
based on the monitored amounts of data traffic, determining an amount of traffic imbalance for the plurality of special-purpose processing units; and
in response to the amount of traffic imbalance exceeding a threshold amount, transmitting a notification indicative of the amount of traffic imbalance.
10. The method of claim 9 , wherein the repeatedly monitoring occurs once per second.
11. The method of claim 9 , further comprising:
determining, based on the monitored amounts of data traffic, respective utilization levels for each of the plurality of special-purpose processing units.
12. An article of manufacture comprising a non-transitory, computer-readable medium having computer-executable instructions thereon that are executable by a processor of an information handling system for:
executing a driver program for controlling a plurality of special-purpose processing units of an information handling system, wherein the driver program is configured to access a counter repeatedly to monitor amounts of data traffic exchanged between respective pairs of the plurality of special-purpose processing units during selected time periods that are determined based on a monitoring policy;
subsequent to each selected time period, causing the counter to be reset;
based on the monitored amounts of data traffic, determining an amount of traffic imbalance for the plurality of special-purpose processing units; and
in response to the amount of traffic imbalance exceeding a threshold amount, transmitting a notification indicative of the amount of traffic imbalance.
13. The article of claim 12 , wherein the special-purpose processing units are graphics processing units (GPUs).
14. The article of claim 12 , wherein the processor is a component of a management controller of the information handling system.
15. The article of claim 12 , wherein the processor is a component of a host system of the information handling system.
16. The article of claim 12 , wherein the instructions are further for:
determining a most-utilized one of the plurality of special-purpose processing units.
17. The article of claim 12 , wherein the instructions are further for:
providing a real-time graphical display indicative of the amounts of data traffic.
18. The article of claim 17 , wherein the real-time graphical display is further indicative of a topology of the plurality of special-purpose processing units.