Apparatus and method with neural network operation
An apparatus and method with neural network operation are provided. A computing apparatus includes one or more processors, storage hardware storing instructions configured to, when executed by the one or more processors, cause the one or more processors to: extract calibration data from training data that is for training a main neural network, based on the calibration data, generate a look up table (LUT) for performing a non-linear function of the main neural network through an auxiliary network corresponding to a layer of the main neural network, and update a parameter of the LUT based on an output of the non-linear function and based on an output of the auxiliary network.
1 . A computing apparatus comprising:
one or more processors;
storage hardware storing instructions configured to, when executed by the one or more processors, cause the one or more processors to:
extract calibration data from training data that is for training a main neural network;
based on the calibration data, generate a look up table (LUT) for performing a non-linear function of the main neural network through an auxiliary network corresponding to a layer of the main neural network, wherein the LUT comprises a first LUT generated based on a first auxiliary network corresponding to a first layer of the main neural network, and wherein the LUT further comprises a second LUT generated based on a second auxillary network corresponding to a second layer of the main neural network; and
update a parameter of the LUT based on an output of the non-linear function and based on an output of the auxiliary network.
2 . The computing apparatus of claim 1 , wherein the calibration data is extracted at a predetermined ratio from the training data.
3 . The computing apparatus of claim 1 , wherein:
a first output of the non-linear function corresponding to a first layer of the main neural network is generated based on the calibration data; and
forward propagation is performed by inputting the first output of the non-linear function to a second layer of the main neural network.
4 . The computing apparatus of claim 1 , wherein the LUT is generated by determining a scale or a bias of the LUT for approximating the non-linear function.
5 . The computing apparatus of claim 1 , wherein a parameter of the auxiliary network is tuned by performing back propagation based on the output of the auxiliary network and the output of the non-linear function.
6 . The computing apparatus of claim 5 , wherein the parameter is tuned based on a mean absolute error between the output of the auxiliary network and the output of the non-linear function.
7 . The computing apparatus of claim 1 , wherein the first layer and the second layer are trained together based on the output of the non-linear function, wherein the non-linear function is an activation function of the first layer and the second layer.
8 . The computing apparatus of claim 1 , wherein the non-linear function comprises a Gaussian error linear unit (GELU) function, a softmax function, a sigmoid function, or a layer normalization function.
9 . A neural network operating method performed by a computing a computing device comprising processing hardware and storage hardware, the method comprising:
extracting calibration data from training data that is for training a main neural network;
generating a look up table (LUT) for approximating a non-linear function of the main neural network, wherein the LUT is generated through an auxiliary network corresponding to a layer of the main neural network based on the calibration data, wherein the LUT comprises a first LUT generated based on a first auxiliary network corresponding to a first layer of the main neural network, and wherein the LUT further comprises a second LUT generated based on a second auxiliary network corresponding to a second layer of the main neural network; and
updating a parameter of the LUT based on an output of the non-linear function and based on an output of the auxiliary network.
10 . The neural network operating method of claim 9 , wherein the LUT is used as an activation function of the main neural network.
11 . The neural network operating method of claim 9 , wherein the updating of the parameter comprises:
generating a first output of the non-linear function corresponding to a first layer of the main neural network, based on the calibration data; and
performing a forward propagation by inputting the first output of the non-linear function to a second layer of the main neural network.
12 . The neural network operating method of claim 9 , wherein the LUT is generated by determining a scale or a bias of the LUT for approximating the non-linear function.
13 . The neural network operating method of claim 9 , wherein the updating of the parameter comprises tuning a parameter of the auxiliary network by performing back propagation based on the output of the auxiliary network and the output of the non-linear function.
14 . The neural network operating method of claim 13 , wherein the parameter is tuned based on a mean absolute error between the output of the auxiliary network and the output of the non-linear function.
15 . The neural network operating method of claim 9 , wherein the updating of the parameter comprises training the first layer and the second layer at the same time based on the output of the non-linear function.
16 . The neural network operating method of claim 9 , wherein the non-linear function comprises a Gaussian error linear unit (GELU) function, a softmax function, a sigmoid function, or a layer normalization function.
17 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 9 .