Neural network processing method and apparatus based on nested bit representation
A neural network processing method and apparatus based on nested bit representation is provided. The processing method includes obtaining first weights for a first layer of a source model of a first layer of a neural network, determining a bit-width for the first layer of the neural network, obtaining second weights for the first layer of the neural network by extracting at least one bit corresponding to the determined bit-width from each of the first weights for the first layer of a source model corresponding to the first layer of the neural network, and processing input data of the first layer of the neural network by executing the first layer of the neural network based on the obtained second weights.
1. A processor-implemented training method, comprising:
executing an iterative process, by one or more processors, with training data and an in-training first neural network, configured to perform a first task, to generate a trained first neural network, configured to perform the first task and a second task different from the first task, that has a trained first layer including trained first weights that each have a first bit-width corresponding to a first precision, the iterative process including:
quantizing in-training first weights, having the first bit-width of an in-training first layer of the in-training first neural network to generate second weights of a first layer of a second neural network, that have a second bit-width that is less than the first bit-width;
executing the second neural network using the second weights, including applying the training data to the first layer of the second neural network and determining loss values, corresponding to the second task, of the first layer of the second neural network;
updating the in-training first weights of the in-training first layer of the in-training first neural network based on the determined loss values; and
performing, for each of the updated in-training first weights, a quantization of a corresponding updated in-training first weight of the updated in-training first weights to generate a corresponding first weight of the trained first weights that includes a nested second weight having the second bit-width that shares bits with the corresponding first weight,
wherein the updating of the in-training first weights comprises updating the in-training first weights of the first bit-width based on statistical information of loss gradients corresponding to the determined loss values,
wherein the updating of the in-training first weights further comprises calculating the statistical information by assigning a high weighted value to a loss gradient corresponding to a weight for which a high priority is set among the second weights of the second bit-width, and
wherein the nested second weight is nested in the corresponding first weight and stored in a same memory space.
2. The method of claim 1 , wherein the executing further includes executing the in-training first neural network using the in-training first weights, including applying the training data to the first layer of the in-training first neural network and determining other loss values, corresponding to the first task, of the first layer of the in-training first neural network, and
wherein the updating of the in-training first weights is further based on the determined other loss values.
3. The method of claim 1 , wherein the quantizing of the in-training first weights includes extracting less that all bits from each of the in-training first weights to generate the second weights.
4. The method of claim 1 , wherein the quantizing of the in-training first weights includes:
determining, for each of the in-training first weights, a corresponding second weight of the second weights to be a corresponding bit group of bits of the second bit-width, included in a corresponding in-training first weight of the in-training first weights, that have lower significance than at least one other bit of the corresponding in-training first weight.
5. The method of claim 2 , wherein the updating of the in-training first weights comprises:
updating the in-training first weights based on respective statistical information of respective loss gradients corresponding to the determined loss values and the determined other loss values.
6. A processor-implemented training method, comprising:
executing an iterative process, by one or more processors, with training data to generate a first neural network, with first layer higher bit-width weights, using lower-precision weights corresponding to an other neural network that are nested within the higher bit-width weights, including:
quantizing weights of a high bit-width, corresponding to a first layer of the first neural network, to generate weights of a low bit-width corresponding to a first layer of the other neural network;
executing the other neural network, including applying the training data to the first layer of the other neural network and determining loss values of the first layer of the other neural network corresponding to the weights of the low bit-width; and
updating the weights of the high bit-width of the first neural network based on the determined loss values,
wherein the updating of the weights of the high bit-width comprises updating the weights of the high bit-width based on statistical information of loss gradients corresponding to the determined loss values,
wherein the updating of the weights of the high bit-width further comprises calculating the statistical information by assigning a high weighted value to a loss gradient corresponding to a weight for which a high priority is set among the weights of the low bit-width, and
wherein the weights of the low bit-width are nested in corresponding higher bit-width and stored in a same memory space.
7. A neural network training apparatus, comprising:
one or more processors configured to execute instructions; and
a memory storing the instructions, that when executed by the one or more processors configure the one or more processors to:
execute an iterative process with training data and an in-training first neural network, configured to perform a first task, to generate a trained first neural network, configured to perform the first task and a second task different from the first task, that has a trained first layer including trained first weights that each have a first bit-width corresponding to a first precision, the iterative including:
a quantizing of in-training first weights, having the first bit-width of an in-training first layer of the in-training first neural network to generate second weights of a first layer of a second neural network, that have a second bit-width that is less than the first bit-width;
an executing of the second neural network using the second weights, including applying the training data to the first layer of the second neural network and determining loss values of the first layer of the second neural network; and
an updating of the in-training first weights of the in-training first layer of the in-training first neural network based on the determined loss values;
a performance, for each of the updated in-training first weights, a quantization of a corresponding updated in-training first weight of the updated in-training first weights to generate a corresponding first weight of the trained first weights that includes a nested second weight having the second bit-width that shares bits with the corresponding first weight,
wherein the updating of the in-training first weights comprises updating the in-training first weights of the first bit-width based on statistical information of loss gradients corresponding to the determined loss values,
wherein the updating of the in-training first weights further comprises calculating the statistical information by assigning a high weighted value to a loss gradient corresponding to a weight for which a high priority is set among the second weights of the second bit-width,
wherein the nested second weight is nested in the corresponding first weight and stored in a same memory space.
8. The training apparatus of claim 7 , wherein the quantizing of the in-training first weights includes extracting less that all bits from each of the in-training first weights to generate the second weights.
9. The training apparatus of claim 7 , wherein the quantizing of the in-training first weights includes determining, for each of the in-training first weights, a corresponding second weight of the second weights to be a corresponding bit group of bits of the second bit-width, included in a corresponding in-training first weight of the in-training first weights, that have lower significance than at least one other bit of the corresponding in-training first weight.