Debugging neural network execution through operation backtracking

1 . A method of partially executing a neural network on a systolic array of a neural network accelerator to debug an execution of the neural network, the method comprising:

executing, a plurality of operations of the neural network on the systolic array to generate, based on an input feature map, an output feature map having a plurality of output values;

determining a difference between an output value of the plurality of output values and a corresponding reference output value generated by a reference processor;

identifying, through operation backtracking, a first set of operations of the plurality of operations that relate a first set of contributing values of a first intermediary feature map to the output value;

identifying, through operation backtracking, a second set of operations of the plurality of operations that relate a second set of contributing values of a second intermediary feature map to the first set of contributing values;

partially executing the neural network on the systolic array by executing each of the plurality of operations except for the first set of operations to generate, based on the input feature map, the first set of contributing values and by executing each of the plurality of operations except for the first set of operations and the second set of operations to generate, based on the input feature map, the second set of contributing values;

partially executing the neural network on the reference processor by executing each of the plurality of operations except for the first set of operations to generate, based on the input feature map, a first set of reference contributing values and by executing each of the plurality of operations except for the first set of operations and the second set of operations to generate a second set of reference contributing values;

performing a first comparison between the first set of contributing values and the first set of reference contributing values;

performing a second comparison between the second set of contributing values and the second set of reference contributing values; and

detecting a computation error related to the execution of the neural network based on the first comparison and the second comparison, wherein the computation error is detected based on one or more of the first set of contributing values differing from one or more of the first set of reference contributing values and one or more of the second set of contributing values not differing from one or more of the second set of reference contributing values.

2 . The method of claim 1 , further comprising:

identifying the first set of contributing values based on the first set of operations; and

identifying the second set of contributing values based on the second set of operations.

3 . The method of claim 2 , wherein the first set of operations include a convolution operation using a filter, and wherein identifying the first set of contributing values is based on at least one of:

a size of the filter;

a filter stride;

a zero-padding of the filter;

a number of input channels; or

a number of output channels.

4 . The method of claim 1 , further comprising:

identifying a first set of contributing processing elements of the systolic array used for the first set of operations; and

identifying a second set of contributing processing elements of the systolic array used for the second set of operations.

5 . The method of claim 1 , further comprising:

determining a cause of the computation error.

6 . A method comprising:

executing a plurality of operations of a neural network on an array of processing elements of an integrated circuit device to generate, based on an input feature map, an output feature map having a plurality of output values;

selecting an output value of the plurality of output values, the output value being erroneous;

identifying a first set of operations of the plurality of operations that relate a first intermediary feature map to the output feature map and a second set of operations of the plurality of operations that relate a second intermediary feature map to the first intermediary feature map;

identifying, based on the first set of operations, a first set of contributing values of the first intermediary feature map that contribute to the output value and, based on the second set of operations, a second set of contributing values of the second intermediary feature map that contribute to the first set of contributing values;

partially executing the neural network on the array of processing elements by executing each of the plurality of operations except for the first set of operations and the second set of operations to generate, based on the input feature map, the second set of contributing values and by executing each of the plurality of operations except for the first set of operations to generate, based on the input feature map, the first set of contributing values;

performing a first comparison between the first set of contributing values and a first set of reference contributing values and a second comparison between the second set of contributing values and a second set of reference contributing values; and

detecting a computation error causing the output value to be erroneous based on the first comparison and the second comparison, wherein the computation error is detected based on one or more of the first set of contributing values differing from one or more of the first set of reference contributing values and one or more of the second set of contributing values not differing from one or more of the second set of reference contributing values.

7 . The method of claim 6 , further comprising:

determining that a difference between the output value and a reference output value exceeds an accuracy threshold, wherein the reference output value is generated using a reference processor based on the input feature map.

8 . The method of claim 7 , wherein the first set of reference contributing values is generated using the reference processor based on the input feature map.

9 . The method of claim 6 , further comprising:

determining a cause of the computation error, wherein the cause of the computation error is either:

a defective processing element of the array of processing elements; or

a summing error due to finite precision.

10 . The method of claim 6 , wherein the first set of operations include a convolution operation using a filter, and wherein identifying the first set of contributing values is based on at least one of:

a size of the filter;

a filter stride;

a zero-padding of the filter;

a number of input channels; or

a number of output channels.

11 . The method of claim 6 , further comprising:

identifying a first set of contributing processing elements of the array of processing elements used for the first set of operations.

12 . A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations comprising:

executing a plurality of operations of a neural network on an array of processing elements of an integrated circuit device to generate, based on an input feature map, an output feature map having a plurality of output values;

selecting an output value of the plurality of output values, the output value being erroneous;

identifying a first set of operations of the plurality of operations that relate a first intermediary feature map to the output feature map and a second set of operations of the plurality of operations that relate a second intermediary feature map to the first intermediary feature map;

identifying, based on the first set of operations, a first set of contributing values of the first intermediary feature map that contribute to the output value and, based on the second set of operations, a second set of contributing values of the second intermediary feature map that contribute to the first set of contributing values;

partially executing the neural network on the array of processing elements by executing each of the plurality of operations except for the first set of operations and the second set of operations to generate, based on the input feature map, the second set of contributing values and by executing each of the plurality of operations except for the first set of operations to generate, based on the input feature map, the first set of contributing values;

performing a first comparison between the first set of contributing values and a first set of reference contributing values and a second comparison between the second set of contributing values and a second set of reference contributing values; and

detecting a computation error causing the output value to be erroneous based on the first comparison and the second comparison, wherein the computation error is detected based on one or more of the first set of contributing values differing from one or more of the first set of reference contributing values and one or more of the second set of contributing values not differing from one or more of the second set of reference contributing values.

13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

determining that a difference between the output value and a reference output value exceeds an accuracy threshold, wherein the reference output value is generated using a reference processor based on the input feature map.

14 . The non-transitory computer-readable medium of claim 13 , wherein the first set of reference contributing values is generated using the reference processor based on the input feature map.

15 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

determining a cause of the computation error is either:

a defective processing element of the array of processing elements; or

a summing error due to finite precision.

16 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise:

identifying a first set of contributing processing elements of the array of processing elements used for the first set of operations.