Training with adaptive runtime and precision profiling
A mechanism is described for facilitating efficient training of neural networks at computing devices. A method of embodiments, as described herein, includes detecting one or more inputs for training of a neural network, and introducing randomness in floating point (FP) numbers to prevent overtraining of the neural network, where introducing randomness includes replacing less-significant low-order bits of operand and result values with new low-order bits during the training of the neural network.
1. An apparatus comprising: a graphics hardware processor to:
detect one or more inputs for a first training of a neural network; track, using a hardware counter, precision of floating-point (FP) values during the first training of the neural network by the hardware counter observing exponents of one or more FP registers of hardware to determine a range of values of the exponents stored in the one or more FP registers, wherein the hardware counter is at least one of continuously or periodically reset;
store, in a precision tracking table based on the tracked precision of FP values, a number of bits that are used to represent single-precision FP value used with each layer of the neural network during the first training of the neural network;
provide automated mixed precision for the neural network by enabling selection of a precision for each layer based on the number of bits stored in the precision tracking table, wherein the precisions selected for each layer can differ from other layers; and
perform a second training of the neural network utilizing the precision selected for each layer of the neural network.
2. The apparatus of claim 1 , wherein the hardware counter is to periodically observe the exponents of the one or more FP registers of the hardware to determine the range of values stored in the one or more FP registers.
3. The apparatus of claim 2 , wherein the hardware counter is implemented using one or more bits based on a range of precision values.
4. The apparatus of claim 1 , wherein an adaptive runtime layer of the graphics hardware processor is to determine a best parallelism from a plurality of parallelisms relating to the neural network on a hardware platform.
5. The apparatus of claim 4 , wherein the plurality of parallelisms comprises one or more of a data parallelism, a model parallelism, and a hybrid parallelism.
6. The apparatus of claim 4 , wherein the adaptive runtime layer is to receive the one or more inputs and is communicatively coupled to one or more of the precision tracking table and a profiling table.
7. The apparatus of claim 1 , wherein the neural network comprises one or more of a convolutional neural network (CNN), a deep neural network (DNN), and a recurrent neural network (RNN), wherein the neural network is based on and in communication with one or more processors including the graphics hardware processor, wherein the graphics hardware processor is co-located with an application processor on a common semiconductor package.
8. A method comprising:
detecting, by a graphics hardware processor, one or more inputs for a first training of a neural network;
tracking, using a hardware counter, precision of floating-point (FP) values during the first training of the neural network by the hardware counter observing exponents of one or more FP registers of hardware to determine a range of values of the exponents stored in the one or more FP registers, wherein the hardware counter is at least one of continuously or periodically reset;
storing, in a precision tracking table based on the tracked precision of FP values, a number of bits that are used to represent single-precision FP value used with each layer of the neural network during the first training of the neural network;
providing automated mixed precision for the neural network by enabling selection of a precision for each layer based on the number of bits stored in the precision tracking table, wherein the precisions selected for each layer can differ from other layers; and
performing a second training of the neural network utilizing the precision selected for each layer of the neural network.
9. The method of claim 8 , wherein the hardware counter is to periodically observe the exponents of the one or more FP registers of the hardware to determine the range of values stored in the one or more FP registers.
10. The method of claim 9 , wherein the hardware counter is implemented using one or more bits based on a range of precision values.
11. The method of claim 8 , wherein an adaptive runtime layer of the graphics hardware processor is to determine a best parallelism from a plurality of parallelisms relating to the neural network on a hardware platform.
12. The method of claim 11 , wherein the plurality of parallelisms comprises one or more of a data parallelism, a model parallelism, and a hybrid parallelism.
13. The method of claim 11 , wherein the adaptive runtime layer is to receive the one or more inputs and is communicatively coupled to one or more of the precision tracking table and a profiling table.
14. The method of claim 8 , wherein the neural network comprises one or more of a convolutional neural network (CNN), a deep neural network (DNN), and a recurrent neural network (RNN), wherein the neural network is based on and in communication with one or more processors including the graphics hardware processor, wherein the graphics hardware processor is co-located with an application processor on a common semiconductor package.
15. At least one non-transitory machine-readable medium comprising instructions that when executed by a computing device, cause the computing device to perform operations comprising:
detecting, by a graphics hardware processor of the computing device, one or more inputs for a first training of a neural network;
tracking, using a hardware counter, precision of floating-point (FP) values during the first training of the neural network by the hardware counter observing exponents of one or more FP registers of hardware to determine a range of values of the exponents stored in the one or more FP registers, wherein the hardware counter is at least one of continuously or periodically reset;
storing, in a precision tracking table based on the tracked precision of FP values, a number of bits that are used to represent single-precision FP value used with each layer of the neural network during the first training of the neural network;
providing automated mixed precision for the neural network by enabling selection of a precision for each layer based on the number of bits stored in the precision tracking table, wherein the precisions selected for each layer can differ from other layers; and
performing a second training of the neural network utilizing the precision selected for each layer of the neural network.
16. The at least one non-transitory machine-readable medium of claim 15 , wherein the hardware counter is to periodically observe the exponents of the one or more FP registers of the hardware to determine the range of values stored in the one or more FP registers.
17. The at least one non-transitory machine-readable medium of claim 16 , wherein the hardware counter is implemented using one or more bits based on a range of precision values.
18. The at least one non-transitory machine-readable medium of claim 15 , wherein an adaptive runtime layer of the graphics hardware processor is to determine a best parallelism from a plurality of parallelisms relating to the neural network on a hardware platform.
19. The at least one non-transitory machine-readable medium of claim 18 , wherein the plurality of parallelisms comprises one or more of a data parallelism, a model parallelism, and a hybrid parallelism.
20. The at least one non-transitory machine-readable medium of claim 18 , wherein the adaptive runtime layer is to receive the one or more inputs and is communicatively coupled to one or more of the precision tracking table and a profiling table, wherein the neural network comprises one or more of a convolutional neural network (CNN), a deep neural network (DNN), and a recurrent neural network (RNN), wherein the neural network is based on and in communication with one or more processors including the graphics hardware processor, wherein the graphics hardware processor is co-located with an application processor on a common semiconductor package.
21. A system comprising: a memory; and a graphics processor communicably coupled to the memory, the graphics processor to:
detect one or more inputs for a first training of a neural network;
track, using a hardware counter, precision of floating-point (FP) values during the first training of the neural network by the hardware counter observing exponents of one or more FP registers of hardware to determine a range of values of the exponents stored in the one or more FP registers, wherein the hardware counter is at least one of continuously or periodically reset;
store, in a precision tracking table based on the tracked precision of FP values, a number of bits that are used to represent single-precision FP value used with each layer of the neural network during the first training of the neural network;
provide automated mixed precision for the neural network by enabling selection of a precision for each layer based on the number of bits stored in the precision tracking table, wherein the precisions selected for each layer can differ from other layers; and
perform a second training of the neural network utilizing the precision selected for each layer of the neural network.
22. The system of claim 21 , wherein the hardware counter is to periodically observe the exponents of the one or more FP registers of the hardware to determine the range of values stored in the one or more FP registers.
23. The system of claim 22 , wherein the hardware counter is implemented using one or more bits based on a range of precision values.