Granted Patent B2

US 12,743,464 · App. 18/378,376 · Granted Sep 22, 2026

Inventors: Xiaoyu Wang (Playa Vista, CA); Ning Xu (Irvine, CA); Ning Zhang (Los Angeles, CA); Vitor Rocha de Carvalho (San Diego, CA); Jia Li (Marina Del Rey, CA)

Assignee: Snap Inc.

G06F16/5866G06F16/587G06F16/9038G06F18/24G06T1/0007G06V10/751G06V10/764G06V10/82H04N23/63G06N5/022G06V2201/09

Abstract

Claims (63)

1 . A system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

accessing image data;

decompressing a first plurality of floating point weights, in response to initiation of processing of the image data;

processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;

decompressing a second plurality of floating point weights, in response to initiation of processing the first intermediate data;

processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and

causing the first intermediate data to be deleted from the system.

2 . The system of claim 1 , wherein the operations further comprise:

receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises:

decompressing the first plurality of floating point weights from the weight indexes.

3 . The system of claim 2 , wherein each weight index of weight indexes indicates a floating point weight of the first plurality of floating point weights.

4 . The system of claim 1 , wherein the system is a mobile device.

5 . The system of claim 1 , wherein the operation further comprise:

capturing, using an image sensor of the system, the image data;

processing the image data as captured by the image sensor to generate a file comprising the image data at a first pixel resolution associated with a pixel height and a pixel width; and

storing the file in a second memory of the system.

6 . The system of claim 5 , wherein the first layer comprises a convolutional layer, and wherein

processing, using the first plurality of floating point weights, the image data comprises: convolving a first kernel with the image data, wherein the first kernel comprises a kernel pixel height less than the pixel height and a kernel pixel width less than the pixel width.

7 . The system of claim 6 , wherein the first intermediate data comprises a plurality of matrixes, each matrix of the plurality of matrixes generated by convolving an associated kernel of a plurality of kernels of the first layer with the image data, wherein the plurality of kernels comprises the first kernel.

8 . The system of claim 1 , wherein the operations further comprise:

generating a plurality of output values from the first output data, each output value of the plurality of output values associated with a corresponding tag of a plurality of tags;

comparing the plurality of output values with a plurality of thresholds, the plurality of thresholds associated with the plurality of tags; and

assigning tags of the plurality of tags if corresponding output values of the plurality of output values transgress corresponding thresholds of the plurality of thresholds.

9 . The system of claim 8 , wherein the operations further comprise:

accessing a plurality of metadata associated with the image data;

processing the assigned tags and the plurality of metadata using a natural language processor to generate extended visual search tags for the image data.

10 . The system of claim 9 further comprising storing the image data and the extended visual search tags, in a memory of the system.

11 . The system of claim 10 further comprising:

receiving, via an input device, a search term; and

generating search results by comparing the search term with the extended visual search tags in the memory of the system.

12 . The system of claim 1 , wherein a first subgraph comprises the first layer of the DCNN and the second layer of the DCNN, and wherein the operations further comprise:

processing the first output data using a first layer of a second subgraph to generate second intermediate data;

processing the second intermediate data using a second layer of the second subgraph to generate second subgraph output data; and

in response to determining that processing of each layer associated with the second intermediate data is completed, causing the second intermediate data to be deleted from the system.

13 . The system of claim 12 , wherein the operations further comprise:

processing the second subgraph output data using a fully connected layer converted to a convolutional layer to generate a dense prediction score map.

14 . The system of claim 13 , wherein the fully connected layer generates the dense prediction score map using an associated output from each convolution layer.

15 . The system of claim 14 , wherein the operations further comprise:

subsampling the dense prediction score map using a max-pooling operating to generate a plurality of output recognition scores, each output recognition score associated with one or more tags.

16 . The system of claim 1 , wherein the first plurality of floating point weights are 16 bit or 32 bit weights.

17 . A non-transitory storage medium comprising instructions that, when executed by one or more processors of a system, cause the system to perform operations, the operations comprising:

accessing image data;

decompressing a first plurality of floating point weights, in response to initiation of processing of the image data;

processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;

decompressing a second plurality of floating point weights, in response to initiation of processing the first intermediate data;

processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and

causing the first intermediate data to be deleted from the system.

18 . The non-transitory storage medium of claim 17 , wherein the operations further comprise:

receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises:

decompressing the first plurality of floating point weights from the weight indexes.

19 . A method for a system, the method comprising:

accessing image data;

decompressing a first plurality of floating point weights;

decompressing a first plurality of floating point weights, in response to initiation of processing of the image data;

processing, using the first plurality of floating point weights, the image data using a first layer of a deep convolutional neural network (DCNN) to generate first intermediate data;

decompressing a second plurality of floating point weights, in response to initiation of processing the first intermediate data;

processing, using the second plurality of floating point weights, the first intermediate data using a second layer of the DCNN to generate first output data; and

causing the first intermediate data to be deleted from the system.

20 . The method of claim 19 further comprising:

receiving a plurality of weights for the DCNN, wherein the plurality of weights are floating point weights compressed to weight indexes, wherein the decompressing the first plurality of floating point weights further comprises:

decompressing the first plurality of floating point weights from the weight indexes.

Assignments (2)

Continuity (6)

References Cited (400)