Content tagging
Systems, methods, devices, media, and computer readable instructions are described for local image tagging in a resource constrained environment. One embodiment involves processing image data using a deep convolutional neural network (DCNN) comprising at least a first subgraph and a second subgraph, the first subgraph comprising at least a first layer and a second layer, processing, the image data using at least the first layer of the first subgraph to generate first intermediate output data; processing, by the mobile device, the first intermediate output data using at least the second layer of the first subgraph to generate first subgraph output data, and in response to a determination that each layer reliant on the first intermediate data have completed processing, deleting the first intermediate data from the mobile device. Additional embodiments involve convolving entire pixel resolutions of the image data against kernels in different layers if the DCNN.
1. A method comprising:
accessing, by one or more processors of a mobile device, image data;
convolving using a first convolution layer, a first plurality of kernels of a deep convolutional neural network (DCNN) with the image data to generate first intermediate data, wherein the first plurality of kernels comprise pixel heights less than a pixel height of the image data and pixel widths less than a pixel width of the image data, and wherein the first plurality of kernels are associated with a first plurality of tags;
convolving using a second convolution layer, a second plurality of kernels of the DCNN with the first intermediate data to generate second intermediate data, wherein the second plurality of kernels are associated with a second plurality of tags;
determining, with a fully connected layer of the DCNN, a third intermediate data from the second intermediate data, the third intermediate data associated with a third plurality of tags; and
assigning one or more tags of the third plurality of tags to portions of the image data based on a comparison of a plurality of output values with threshold values, wherein each tag of the first plurality of tags, the second plurality of tags, and the third plurality of tags indicates a prediction score map for an image category.
2. The method of claim 1 wherein the method further comprises:
capturing, using an image sensor of the mobile device, the image data.
3. The method of claim 1 further comprising:
converting the fully connected layer of the DCNN to a third convolution layer.
4. The method of claim 3 wherein the third convolution layer comprises: a third plurality of kernels of the DCNN and wherein the third plurality of kernels are associated with the third plurality of tags.
5. The method of claim 1 wherein the first convolution layer and the second convolution layer convolve across the pixel width of the image data and the pixel height of the image data.
6. The method of claim 1 wherein the image data is not split into multiple sub-windows.
7. The method of claim 1 wherein the DCNN comprises additional convolution layers.
8. The method of claim 1 wherein 16-bit half precision values are used to store first intermediate data and second intermediate data.
9. The method of claim 1 further comprising: receiving, by one or more processors of the mobile device, a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indices.
10. The method of claim 9 further comprising:
in response to convolving processing, using the first convolution layer, decompressing a first set of weight indices.
11. The method of claim 1 wherein convolving, using the first convolution layer, further comprises: using a corresponding first set of floating point weights as decompressed from weight indices.
12. The method of claim 1 wherein the DCNN comprises only convolutional layers and a fully connected layer.
13. The method of claim 1 wherein the DCNN uses a plurality of weights comprising 16 bit weights, 32 bit weights, or 64 bit weights.
14. The method of claim 1 wherein the DCNN refrains from using a max-pooling layer.
15. The method of claim 1 further comprising:
storing the image data and indications of the one or more assigned tags to portions of the image data.
16. A mobile device for image tagging comprising: a memory; an image sensor coupled to the memory; and one or more processors coupled to the memory and configured to:
access, by one or more processors of a mobile device, image data;
convolve using a first convolution layer, a first plurality of kernels of a deep convolutional neural network (DCNN) with the image data to generate first intermediate data, wherein the first plurality of kernels comprise pixel heights less than a pixel height of the image data and pixel widths less than a pixel width of the image data, and wherein the first plurality of kernels are associated with a first plurality of tags;
convolve using a second convolution layer, a second plurality of kernels of the DCNN with the first intermediate data to generate second intermediate data, wherein the second plurality of kernels are associated with a second plurality of tags;
determine, with a fully connected layer of the DCNN, a third intermediate data from the second intermediate data, the third intermediate data associated with a third plurality of tags; and
assign one or more tags of the third plurality of tags to portions of the image data based on a comparison of a plurality of output values with threshold values, wherein each tag of the first plurality of tags, the second plurality of tags, and the third plurality of tags indicates a prediction score map for an image category.
17. A non-transitory storage medium comprising instructions that, when executed by one or more processors of a mobile device, cause the mobile device to perform operations for local image tagging, the operations comprising:
accessing, by one or more processors of a mobile device, image data;
convolving using a first convolution layer, a first plurality of kernels of a deep convolutional neural network (DCNN) with the image data to generate first intermediate data, wherein the first plurality of kernels comprise pixel heights less than a pixel height of the image data and pixel widths less than a pixel width of the image data, and wherein the first plurality of kernels are associated with a first plurality of tags;
convolving using a second convolution layer, a second plurality of kernels of the DCNN with the first intermediate data to generate second intermediate data, wherein the second plurality of kernels are associated with a second plurality of tags;
determining, with a fully connected layer of the DCNN, a third intermediate data from the second intermediate data, the third intermediate data associated with a third plurality of tags; and
assigning one or more tags of the third plurality of tags to portions of the image data based on a comparison of a plurality of output values with threshold values, wherein each tag of the first plurality of tags, the second plurality of tags, and the third plurality of tags indicates a prediction score map for an image category.