Quantization and inferencing for low-bitwidth neural networks
A method for operating a low-bitwidth neural network includes converting a first activation to a non-negative value (e.g., absolute value). The first activation has a signed value. The sign of the activation is used to select a weight value. A product of the non-negative activation and the selected weight value is computed to determine a next activation. The next activation is quantized and supplied to a subsequent layer of the low-bitwidth neural network.
1. A method for operating a low-bitwidth neural network, comprising:
converting a first activation to a non-negative value of the first activation, the first activation having a sign value;
selecting a weight value;
computing a product of the non-negative value of the first activation and the selected weight value to determine a next activation; and
quantizing the next activation, the next activation being supplied to a subsequent layer of the low-bitwidth neural network.
2. The method of claim 1 , further comprising:
extracting sign bits of the first activation, the sign bits corresponding to a sign of the first activation; and
wherein the weight value is selected based at least in part on the sign of the first activation.
3. The method of claim 1 , in which the first activation is a two bit value and the weight value is a one bit value.
4. The method of claim 1 , further comprising computing the product by a bitwise exclusive OR (XOR) function based on the non-negative value of the first activation and the selected weight value.
5. The method of claim 1 , further comprising training the weight value for positive or negative activations using a phase selective convolution technique.
6. The method of claim 1 , in which the next activation is represented as a non-negative value of the next activation, sign bits of the next activation being extracted and supplied to a next layer with the non-negative value of the next activation.
7. The method of claim 1 , in which the non-negative value of the first activation comprises an absolute value of the first activation.
8. An apparatus for operating a low-bitwidth neural network, comprising:
a memory; and
at least one processor coupled to the memory, the at least one processor being configured:
to convert a first activation to a non-negative value of the first activation, the first activation having a sign value;
to select a weight value;
to compute a product of the non-negative value of the first activation and the selected weight value to determine a next activation; and
to quantize the next activation, the next activation being supplied to a subsequent layer of the low-bitwidth neural network.
9. The apparatus of claim 8 , in which the at least one processor is further configured:
to extract sign bits of the first activation, the sign bits corresponding to a sign of the first activation; and
to select the weight value based at least in part on the sign of the first activation.
10. The apparatus of claim 8 , in which the first activation is a two bit value and the weight value is a one bit value.
11. The apparatus of claim 8 , in which the at least one processor is further configured to compute the product by a bitwise exclusive OR (XOR) function based on the non-negative value of the first activation and the selected weight value.
12. The apparatus of claim 8 , in which the at least one processor is further configured to train the weight value for positive or negative activations using a phase selective convolution technique.
13. The apparatus of claim 8 , in which the at least one processor is further configured:
to represent the next activation as the non-negative value of the next activation; and
to extract sign bits of the next activation and to supply the extracted sign bits to a next layer with the non-negative value of the next activation.
14. The apparatus of claim 8 , in which the non-negative value of the first activation comprises an absolute value of the first activation.
15. An apparatus for operating a low-bitwidth neural network, comprising:
means for converting a first activation to a non-negative value of the first activation, the first activation having a sign value;
means for selecting a weight value;
means for computing a product of the non-negative value of the first activation and the selected weight value to determine a next activation; and
means for quantizing the next activation, the next activation being supplied to a subsequent layer of the low-bitwidth neural network.
16. The apparatus of claim 15 , further comprising:
means for extracting sign bits of the first activation, the sign bits corresponding to a sign of the first activation; and
means for selecting the weight value based at least in part on the sign of the first activation.
17. The apparatus of claim 15 , in which the first activation is a two bit value and the weight value is a one bit value.
18. The apparatus of claim 15 , further comprising means for computing the product by a bitwise exclusive OR (XOR) function based on the non-negative value of the first activation and the selected weight value.
19. The apparatus of claim 15 , further comprising means for training the weight value for positive or negative activations using a phase selective convolution technique.
20. The apparatus of claim 15 , further comprising means for representing the next activation as the non-negative value of the next activation, and
means for extracting sign bits of the next activation and
means for supplying the extracted sign bits to a next layer with the non-negative value of the next activation.