System and methods for efficiently implementing a convolutional neural network incorporating binarized filter and convolution operation for performing image classification
Systems, apparatuses, and methods for efficiently and accurately processing an image in order to detect and identify one or more objects contained in the image, and methods that may be implemented on mobile or other resource constrained devices. Embodiments of the invention introduce simple, efficient, and accurate approximations to the functions performed by a convolutional neural network (CNN); this is achieved by binarization (i.e., converting one form of data to binary values) of the weights and of the intermediate representations of data in a convolutional neural network. The inventive binarization methods include optimization processes that determine the best approximations of the convolution operations that are part of implementing a CNN using binary operations.
1. A method for processing an image, comprising:
by a camera of a mobile computing device, capturing the image;
by a processor of the mobile computing device, inputting data representing the image into a convolutional neural network (CNN), the CNN including a plurality of convolutional layers, a set of weights or filters for at least one of the layers, and a set of input data to the at least one of the layers;
by the processor of the mobile computing device, representing a convolution operation between the set of input data and the set of filters or weights by a product of a scaling factor and a binary representation of the set of filters or weights convolved with the set of input data, wherein the binary representation is the sign of the weight values, and the scaling factor is the average of the absolute weight values; and
by the processor of the mobile computing device, applying a classification operation to an output of the last of the plurality of convolutional layers.
2. The method of claim 1 , further comprising representing the set of input data by a product of a scaling factor and a binary representation of the input data.
3. The method of claim 2 , wherein the binary representation of the set of input data is the sign of the input data values, and the scaling factor is the average of the absolute input data values.
4. The method of claim 3 , wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.
5. The method of claim 1 , further comprising processing the data representing the image prior to inputting the data into the convolutional neural network, wherein the processing the data includes one or more of reducing noise in the data, scaling the data, or sampling the data.
6. The method of claim 1 , wherein the scaling factor is determined by solving an optimization problem of the form given by:
J
(
B
,
α
)
=
W
-
α
B
2
α
*
,
B
*
=
argmin
α
,
B
,
J
(
B
,
α
)
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.
7. One or more computer-readable non-transitory storage media embodying software comprising instructions operable when executed to:
by a camera of a mobile computing device, capture an image;
by a processor of the mobile computing device, input data representing the image into a convolutional neural network (CNN), the CNN including a plurality of convolutional layers, a set of weights or filters for at least one of the layers, and a set of input data to the at least one of the layers;
by the processor of the mobile computing device, represent a convolution operation between the set of input data and the set of filters or weights by a product of a scaling factor and a binary representation of the set of input data convolved with the set of filters or weights, wherein the binary representation is the sign of the weight values, and the scaling factor is the average of the absolute weight values; and
by the processor of the mobile computing device, apply a classification operation to an output of the last of the plurality of convolutional layers.
8. The computer-readable non-transitory storage media of claim 7 , further comprising representing the set of filters or weights by a product of a scaling factor and a binary representation of the set of filters or weights.
9. The computer-readable non-transitory storage media of claim 8 , wherein the binary representation of the set of filters or weights is the sign of the weight values, and the scaling factor is the average of the absolute weight values.
10. The computer-readable non-transitory storage media of claim 9 , wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.
11. The computer-readable non-transitory storage media of claim 7 , further comprising processing the data representing the image prior to inputting the data into the convolutional neural network, wherein the processing the data includes one or more of reducing noise in the data, scaling the data, or sampling the data.
12. The computer-readable non-transitory storage media of claim 7 , wherein the scaling factor is determined by solving an optimization problem of the form given by:
J
(
B
,
α
)
=
W
-
α
B
2
α
*
,
B
*
=
argmin
α
,
B
,
J
(
B
,
α
)
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.
13. A mobile device for processing an image, comprising:
a camera of the mobile computing device for capturing the image;
a memory of the mobile computing device for storing an ordered set of processing blocks, wherein the ordered set includes a normalization block, an activation block, a convolution block, and a pooling block, with image input data being processed in that order;
a processor of the mobile computing device for executing a computer-implemented process for representing the convolution block, wherein the computer-implemented process represents a convolution operation between a set of data and a set of filters or weights by a product of a scaling factor and a binary representation of the set of filters or weights convolved with the set of data, wherein the binary representation is the sign of the weight values, and the scaling factor is the average of the absolute weight values; and
a processor of the mobile computing device for applying a classifier to an output of the pooling block.
14. The mobile device of claim 13 , wherein the processor further executes a computer-implemented process for representing the data by a product of a scaling factor and a binary representation of the data.
15. The mobile device of 14 , wherein the convolution operation is implemented by a combination of an XNOR operation and a bit-counting operation.
16. The mobile device of claim 14 , wherein the binary representation of the set of data is the sign of the data values, and the scaling factor is the average of the absolute data values.
17. The mobile device of claim 13 , wherein the scaling factor is determined by solving an optimization problem of the form given by:
J
(
B
,
α
)
=
W
-
α
B
2
α
*
,
B
*
=
argmin
α
,
B
,
J
(
B
,
α
)
where α is the scaling factor, B is a binary representation of the set of filters or weights, and W is the set of filters or weights.