Lookup-based convolutional neural network

1. A system comprising:

an image sensor configured to capture images;

a fastening article attached to the image sensor and configured to mount the image sensor in proximity to a portion of a human body; and

a processing apparatus that is configured to:

access an image captured using the image sensor;

apply a convolutional neural network to image data based on the image to obtain an output, in which a layer of the convolutional network includes filters with weights that are stored as a dictionary (D) of channel weight vectors, a respective lookup index tensor (I) that indexes the dictionary, and a respective lookup coefficient tensor (C), and in which applying the convolutional neural network includes:

convolving the channel weight vectors of the dictionary (D) with an input tensor based on the image to obtain an input dictionary (S), and

combining entries of the input dictionary (S) that are indexed with indices from the respective lookup index tensor (I) and multiplied with corresponding coefficients from the respective lookup coefficient tensor (C); and

store, display, or transmit data based on the output of the convolutional neural network.

2. The system of claim 1 , in which the processing apparatus is configured to train the convolutional neural network by back-propagation using a sparse tensor (P), wherein non-zero entries of the sparse tensor (P) correspond to coefficients of the respective lookup coefficient tensor (C) at locations corresponding to indices of the respective lookup index tensor (I), and in which performing a training iteration comprises:

convolving the channel weight vectors of the dictionary (D) with a training input tensor based on a training image to obtain a training input dictionary (S);

convolving the training input dictionary (S) with the sparse tensor (P) to obtain training activations;

determining a loss function based on the training activations; and

updating the sparse tensor (P) based on the loss function.

3. The system of claim 1 , in which the processing apparatus is configured to train the convolutional neural network using back-propagation to update the respective lookup index tensor (I) and the respective lookup coefficient tensor (C) in multiple training iterations, while holding the dictionary (D) of channel weight vectors fixed.

4. The system of claim 1 , in which the processing apparatus is configured to determine the dictionary (D) of channel weight vectors by training a different neural network that includes the dictionary (D), wherein the different neural network has less layers than the convolutional neural network.

5. A method comprising:

applying a convolutional neural network to image data based on an image to obtain an output, in which a layer of the convolutional network includes filters with weights that are stored as a dictionary (D) of channel weight vectors, a respective lookup index tensor (I) that indexes the dictionary, and a respective lookup coefficient tensor (C), and in which applying the convolutional neural network includes:

convolving the channel weight vectors of the dictionary (D) with an input tensor based on the image to obtain an input dictionary (S), and

combining entries of the input dictionary (S) that are indexed with indices from the respective lookup index tensor (I) and multiplied with corresponding coefficients from the respective lookup coefficient tensor (C); and

storing, displaying, or transmitting data based on the output of the convolutional neural network.

6. The method of claim 5 , comprising train the convolutional neural network by back-propagation using a sparse tensor (P), wherein non-zero entries of the sparse tensor (P) correspond to coefficients of the respective lookup coefficient tensor (C) at locations corresponding to indices of the respective lookup index tensor (I), and in which performing a training iteration comprises:

convolving the channel weight vectors of the dictionary (D) with a training input tensor based on a training image to obtain a training input dictionary (S);

convolving the training input dictionary (S) with the sparse tensor (P) to obtain training activations;

determining a loss function based on the training activations; and

updating the sparse tensor (P) based on the loss function.

7. The method of claim 6 , in which the loss function includes a regularization term based on a norm of the sparse tensor (P).

8. The method of claim 6 , in which the training iteration comprises:

sorting entries of a vector of the sparse tensor (P) corresponding to a spatial location; and

zeroing out all but the s largest entries of the vector of the sparse tensor (P), wherein s is a sparsity constraint.

9. The method of claim 6 , in which the training iteration comprises:

applying a threshold function to entries of the sparse tensor (P).

10. The method of claim 6 , in which the training iteration comprises:

back-propagating through a threshold function to determine a gradient with respect to the sparse tensor (P).

11. The method of claim 5 , comprising training the convolutional neural network using back-propagation to update the respective lookup index tensor (I) and the respective lookup coefficient tensor (C) in multiple training iterations, while holding the dictionary (D) of channel weight vectors fixed.

12. The method of claim 5 , in which the dictionary (D) of channel weight vectors is used in multiple layers of the convolutional neural network.

13. The method of claim 5 , in which the dictionary (D) of channel weight vectors has been determined by training a different neural network that includes the dictionary (D), wherein the different neural network is a different depth than the convolutional neural network.

14. The method of claim 5 , comprising determining the dictionary (D) of channel weight vectors by training a different neural network that includes the dictionary (D), wherein the different neural network has less layers than the convolutional neural network.

15. The method of claim 5 , in which the output is a classification of the image.

16. A wearable device comprising:

a memory; and

a processor, wherein the memory stores instructions executable by the processor to cause the wearable device to:

apply a convolutional neural network to image data based on an image to obtain an output, in which a layer of the convolutional network includes filters with weights that are stored as a dictionary (D) of channel weight vectors, a respective lookup index tensor (I) that indexes the dictionary, and a respective lookup coefficient tensor (C), and in which applying the convolutional neural network includes:

convolving the channel weight vectors of the dictionary (D) with an input tensor based on the image to obtain an input dictionary (S), and

combining entries of the input dictionary (S) that are indexed with indices from the respective lookup index tensor (I) and multiplied with corresponding coefficients from the respective lookup coefficient tensor (C); and

store, display, or transmit data based on the output of the convolutional neural network.

17. The wearable device of claim 16 , in which the memory stores instructions executable by the processor to cause the wearable device to train the convolutional neural network by back-propagation using a sparse tensor (P), wherein non-zero entries of the sparse tensor (P) correspond to coefficients of the respective lookup coefficient tensor (C) at locations corresponding to indices of the respective lookup index tensor (I), and in which performing a training iteration comprises:

convolving the channel weight vectors of the dictionary (D) with a training input tensor based on a training image to obtain a training input dictionary (S);

convolving the training input dictionary (S) with the sparse tensor (P) to obtain training activations;

determining a loss function based on the training activations; and

updating the sparse tensor (P) based on the loss function.

18. The wearable device of claim 16 , in which the memory stores instructions executable by the processor to cause the wearable device to train the convolutional neural network using back-propagation to update the respective lookup index tensor (I) and the respective lookup coefficient tensor (C) in multiple training iterations, while holding the dictionary (D) of channel weight vectors fixed.

19. The wearable device of claim 16 , in which the dictionary (D) of channel weight vectors is used in multiple layers of the convolutional neural network.

20. The wearable device of claim 16 , in which the memory stores instructions executable by the processor to cause the wearable device to determine the dictionary (D) of channel weight vectors by training a different neural network that includes the dictionary (D), wherein the different neural network has less layers than the convolutional neural network.