Sub-graph in frequency domain and dynamic selection of convolution implementation on a GPU
In an example, an apparatus comprises a plurality of execution units; and logic, at least partially including hardware logic, to determine a sub-graph of a network that can be executed in a frequency domain and apply computations in the sub-graph in the frequency domain. Other embodiments are also disclosed and claimed.
1. An apparatus comprising processing circuitry to:
obtain a sub-graph of a convolutional neural network to be executed at least in part in a frequency domain;
generate a predicted level of activation sparsity for one or more layers of the convolutional neural network; and
apply one or more convolutional computations in the sub-graph in the frequency domain, wherein the one or more convolutional computations are performed at variable levels of integer precisions based at least in part on the predicted level of activation sparsity of a layer in the convolutional neural network using a first set of execution resources, while internal computations are performed in a baseline precision level of 8-bits or 16-bits using a second set of execution resources, different from the first set of execution resources, wherein:
the first set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations; and
the second set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations.
2. The apparatus of claim 1 , the processing circuitry to:
dynamically select a convolutional implementation based at least in part on executing a short comparison of one or more convolutions in the convolutional neural network.
3. The apparatus of claim 1 , the processing circuitry to:
generate a predicted level of activation sparsity for one or more layers of the convolutional neural network using training sample statistics for the convolutional neural network.
4. The apparatus of claim 3 , the processing circuitry to:
update the predicted level of activation sparsity when operating on the convolutional neural network in inference mode.
5. The apparatus of claim 1 , the processing circuitry to:
expose one or more embedded cast operations in a load/store instruction to support loading data for the one or more convolutional computations in a variable integer precision.
6. The apparatus of claim 5 , the processing circuitry to:
select between a 2-bit precision, a 3-bit precision, and a 7-bit precision level.
7. A method, comprising:
obtaining a sub-graph of a convolutional neural network to be executed at least in part in a frequency domain;
generating a predicted level of activation sparsity for one or more layers of the convolutional neural network; and
applying one or more convolutional computations in the sub-graph in the frequency domain, wherein the one or more convolutional computations are performed at variable levels of integer precisions based at least in part on the predicted level of activation sparsity of a layer in the convolutional neural network using a first set of execution resources, while internal computations are performed in a baseline precision level of 8-bits or 16-bits using a second set of execution resources, different from the first set of execution resources, wherein:
the first set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations; and
the second set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations.
8. The method of claim 7 , further comprising:
dynamically selecting a convolutional implementation based at least in part on executing a short comparison of one or more convolutions in the convolutional neural network.
9. The method of claim 7 , further comprising:
generating a predicted level of activation sparsity for one or more layers of the convolutional neural network using training sample statistics for the convolutional neural network.
10. The method of claim 9 , further comprising:
updating the predicted level of activation sparsity when operating on the convolutional neural network in inference mode.
11. The method of claim 7 , further comprising:
exposing one or more embedded cast operations in a load/store instruction to support loading data for the one or more convolutional computations in a variable integer precision.
12. The method of claim 7 , further comprising:
selecting between a 2-bit precision, a 3-bit precision, and a 7-bit precision level.
13. One or more non-transitory computer readable media comprising instructions which, when executed by processing circuitry, configure the processing circuitry to:
obtain a sub-graph of a convolutional neural network to be executed at least in part in a frequency domain;
generate a predicted level of activation sparsity for one or more layers of the convolutional neural network; and
apply one or more convolutional computations in the sub-graph in the frequency domain, wherein the one or more convolutional computations are performed at variable levels of integer precisions based at least in part on the predicted level of activation sparsity of a layer in the convolutional neural network using a first set of execution resources, while internal computations are performed in a baseline precision level of 8-bits or 16-bits using a second set of execution resources, different from the first set of execution resources, wherein:
the first set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations; and
the second set of execution resources comprises a first plurality of processing elements configured to perform dense convolution operations.
14. The one or more non-transitory computer readable media of claim 13 , further comprising instructions which, when executed by processing circuitry, configure the processing circuitry to:
dynamically select a convolutional implementation based at least in part on executing a short comparison of one or more convolutions in the convolutional neural network.
15. The one or more non-transitory computer readable media of claim 13 , further comprising instructions which, when executed by processing circuitry, configure the processing circuitry to:
generate a predicted level of activation sparsity for one or more layers of the convolutional neural network using training sample statistics for the convolutional neural network.
16. The one or more non-transitory computer readable media of claim 15 , further comprising instructions which, when executed by processing circuitry, configure the processing circuitry to:
update the predicted level of activation sparsity when operating on the convolutional neural network in inference mode.
17. The one or more non-transitory computer readable media of claim 13 , further comprising instructions which, when executed by processing circuitry, configure the processing circuitry to:
expose one or more embedded cast operations in a load/store instruction to support loading data for the one or more convolutional computations in a variable integer precision.