Granted Patent B2

US 12,619,515 · App. 18/085,220 · Granted May 5, 2026

Inventors: Satyanarayana Raju Uppalapati (Hyderabad, IN); Rajasekhar Reddy Ereddy (Hyderabad, IN); Sameek Banerjee (Hyderabad, IN); Mohammed Shahim (Thrissur, IN); Shilpa Kallem (Hyderabad, IN); Suresh Kumar Vennam (Hyderabad, IN); Abhilash Bharath Ghanore (Hyderabad, IN); Raju Datla (Hyderabad, IN); Wajahat Qadeer (Los Altos, CA); Rehan Hameed (Campbell, CA)

Assignee: Deep Vision Inc.

G06F11/3452G06F11/3409G06N3/045G06F11/3447

Abstract

Claims (97)

1 . A method for profiling neural network performance, the method comprising:

accessing a static schedule of a target neural network for execution by a processing device, the target neural network comprising a set of layers defining a set of operations;

generating a set of expected performance metrics for the target neural network based on the static schedule, the set of expected performance metrics comprising a subset of expected performance metrics for each layer in the set of layers;

activating a set of performance monitor registers in the processing device based on signals stored in a memory of the processing device;

capturing, by the set of performance monitor registers, a set of runtime performance metrics during execution of the target neural network by the processing device;

for each layer in the set of layers:

associating a subset of runtime performance metrics, in the set of runtime performance metrics, with the layer based on detection of an identifier assigned to the layer during execution of the layer;

calculating a first difference between a runtime performance metric, in the subset of runtime performance metrics for the layer, and a corresponding estimated performance metric in the subset of estimated performance metrics for the layer; and

in response to detecting the first difference exceeding a first threshold, identifying the layer as a deviant layer in a subset of deviant layers in the set of layers; and

serving, at a user interface, a notification indicating the subset of deviant layers.

2 . The method of claim 1 :

wherein generating the set of expected performance metrics comprises generating the set of expected performance metrics further comprising a second subset of expected performance metrics for each operation in the set of operations;

wherein generating the set of runtime performance metrics comprises generating the set of runtime performance metrics further comprising a second subset of runtime performance metrics for each operation in the set of operations; and

further comprising:

for each operation in the set of operations:

calculating a second difference between a runtime performance metric, in the second subset of runtime performance metrics for the operation, and a corresponding expected performance metric in the second subset of estimated performance metrics for the operation; and

in response to detecting the second difference exceeding a second threshold, identifying the operation as a deviant operation in a subset of deviant operations in the set of operations; and

serving, at the user interface, a notification indicating the subset of deviant operations.

3 . The method of claim 1 :

wherein capturing the set of runtime performance metrics comprises capturing the set of runtime performance metrics comprising a first runtime metric representing a first number of compute cycles per instruction in a first layer in the set of layers; and

further comprising:

in response to detecting the first number of compute cycles per instruction in the first layer exceeding a second threshold, identifying the first layer as a compute-bound layer in a subset of compute-bound layers in the set of layers; and

serving, at the user interface, a notification indicating the subset of compute-bound layers.

4 . The method of claim 1 :

wherein capturing the set of runtime performance metrics comprises capturing the set of runtime performance metrics comprising a first runtime metric representing a first number of data movement cycles per instruction in a first layer in the set of layers; and

further comprising:

in response to detecting the first number of data movement cycles per instruction in the first layer exceeding a second threshold, identifying the first layer as a bandwidth-bound layer in a subset of bandwidth-bound layers in the set of layers; and

serving, at the user interface, a notification indicating the subset of bandwidth-bound layers.

5 . The method of claim 1 , further comprising:

generating a first execution timeline representing expected execution of the target neural network based on the static schedule and the set of expected performance metrics;

generating a second execution timeline representing execution of the target neural network by the processing device based on the set of runtime performance metrics; and

serving, at the user interface, a visualization depicting the first execution timeline and the second execution timeline.

6 . The method of claim 1 :

further comprising accessing a cost model defining a set of costs for each operation in the set of operations; and

wherein generating the set of expected performance metrics comprises generating the set of expected performance metrics based on the static schedule and the cost model.

7 . The method of claim 1 :

wherein accessing the static schedule comprises accessing the static schedule assigning a set of nodes, defining the set of operations in a directed acyclic graph, to a set of resources in the processing device;

wherein capturing the set of runtime performance metrics comprises capturing the set of runtime performance metrics comprising a second runtime performance metric representing memory bandwidth utilization for a first node in the set of nodes, the first node comprising a subset of operations in the set of operations; and

further comprising serving, at the user interface, a notification indicating the second runtime performance metric.

8 . The method of claim 1 , wherein activating the set of performance monitor registers comprises:

writing a set of signals to a set of memory regions, in the memory of the processing device, mapped to the set of performance monitor registers; and

selectively activating the set of performance monitor registers based on the set of signals stored in the set of memory regions.

9 . The method of claim 1 :

wherein capturing the set of runtime performance metrics comprises capturing the set of runtime performance metrics further comprising:

a second runtime performance metric representing a computational cost of a first layer in the set of layers; and

a third runtime performance metric representing a memory bandwidth cost of the first layer; and

further comprising serving, at the user interface, a notification indicating the second runtime performance metric and the third runtime performance metric.

10 . The method of claim 9 , wherein capturing the set of runtime performance metrics comprises generating the second runtime performance metric based on a second subset of runtime performance metrics in the set of runtime performance metrics, the second subset of runtime performance metrics representing computation costs of a first subset of operations defined by the first layer.

11 . The method of claim 1 , wherein associating the subset of runtime performance metrics for each layer in the set of layers comprises:

in response to detecting a change in execution from the layer to a succeeding layer in the set of layers, extracting the subset of runtime performance metrics from the set of performance monitor registers; and

transferring the subset of runtime performance metrics to a main memory of the processing unit via a direct memory access core.

12 . The method of claim 11 , wherein extracting the subset of runtime performance metrics from the set of performance monitor registers comprises:

detecting the change in execution from the layer to the succeeding layer by:

reading a first set of bits corresponding to the identifier assigned to the layer in a first dequeued command in a control processor of the processing device; and

reading a second set of bits corresponding to a second identifier assigned to the succeeding layer in a second dequeued command in the control processor; and

in response to detecting the change in execution from the layer to the succeeding layer, extracting the subset of runtime performance metrics from the set of performance monitor registers.

13 . A method for profiling neural network performance, the method comprising:

accessing a first static schedule of a target neural network for execution by a first processing device comprising a first set of resources, the target neural network comprising a set of layers defining a set of operations;

generating a first set of expected performance metrics for the target neural network based on the first static schedule, the first set of expected performance metrics comprising a first expected performance metric for a first layer in the set of layers; and

during execution of the target neural network by the first processing device:

activating a set of performance monitor registers in the processing device based on signals stored in a memory of the processing device;

capturing, by the set of performance monitor registers, a first set of runtime performance metrics for the target neural network;

associating a first runtime performance metric, in the first set of runtime performance metrics, with the first layer based on detection of a first identifier assigned to the first layer during execution of the first layer;

in response to detecting a difference between the first runtime performance metric and the first expected performance metric exceeding a first threshold, identifying the first layer as a deviant layer in a subset of deviant layers in the set of layers; and

serving, at a user interface, a notification indicating the subset of deviant layers.

14 . The method of claim 13 :

wherein capturing the first set of runtime performance metrics comprises capturing the first set of runtime performance metrics comprising a second runtime performance metric representing a first number of inferences per second resulting from execution of the target neural network by the first processing device; and

further comprising serving, at the user interface, a notification indicating the second runtime performance metric.

15 . The method of claim 14 :

wherein accessing the first static schedule comprises accessing the first static schedule of the target neural network for execution by the first processing device characterized by a first device type; and

further comprising:

accessing a second static schedule of the target neural network for execution by a second processing device characterized by a second device type different from the first device type;

generating a second set of expected performance metrics for the target neural network based on the second static schedule;

generating a second set of runtime performance metrics during execution of the target neural network by the second processing device, the second set of runtime performance metrics comprising a third runtime performance metric representing a second number of inferences per second resulting from execution of the target neural network by the second processing device; and

in response to detecting the first number of inferences per second exceeding the second number of inferences per second, serving, at the user interface, a notification indicating the first processing device as a higher performing processing device for the target neural network than the second processing device.

16 . The method of claim 13 :

wherein generating the first set of expected performance metrics comprises generating the first set of expected performance metrics comprising a second estimated performance metric representing an expected accuracy of the first layer in the set of layers;

wherein capturing the first set of performance metrics comprises capturing a set of saturation counts by a set of saturation counters in the set of performance monitor registers;

wherein capturing the first set of runtime performance metrics comprises generating a second runtime performance metric, in the first set of runtime performance metrics, based on the set of saturation counts, the second runtime performance metric representing an accuracy of the first layer during execution of the target neural network by the first processing device; and

further comprising, in response to detecting a difference between the second runtime performance metric and the second expected performance metric exceeding a first threshold, serving an alert at the user interface.

17 . The method of claim 13 :

wherein capturing the first set of runtime performance metrics comprises capturing the first set of runtime performance metrics comprising a second runtime performance metric representing utilization of a first resource in the first set of resources; and

further comprising, in response to detecting the second runtime performance metric exceeding a second threshold, serving an alert at the user interface.

18 . The method of claim 13 , further comprising writing, to a first memory region mapped to a first performance monitor register in the set of performance monitor registers, a signal representing a command to disable the first performance monitor register.

19 . The method of claim 13 :

wherein capturing the first set of runtime performance metrics comprises capturing the first set of runtime performance metrics comprising a second runtime performance metric representing utilization of the first processing device; and

further comprising, in response to detecting the second runtime performance metric exceeding a second threshold, load balancing execution of the target neural network between the first processing device and a second processing device.

20 . A method for profiling neural network performance, the method comprising:

accessing a static schedule of a target neural network for execution by a first processing device, the target neural network comprising a set of layers;

generating a set of expected performance metrics of the target neural network based on the static schedule, the set of expected performance metrics comprising a first expected performance metric for a first layer in the set of layers;

activating a set of performance monitor registers in the first processing device based on signals stored in a memory of the first processing device;

capturing, by the set of performance monitor registers, a set of runtime performance metrics during execution of the target neural network by the first processing device, the set of runtime performance metrics comprising:

a first runtime performance metric for the first layer; and

a second runtime performance metric representing utilization of the first processing device;

associating the first runtime performance metric with the first layer based on detection of a first identifier assigned to the first layer during execution of the first layer;

in response to detecting a difference between the first runtime performance metric and the first expected performance metric exceeding a threshold, identifying the first layer as a deviant layer in a subset of deviant layers in the set of layers; and

serving, at a user interface, a notification indicating the subset of deviant layers.

Assignments (2)

Continuity (2)

References Cited (54)