Granted Patent B2

US 12,749,296 · App. 17/970,907 · Granted Sep 29, 2026

Inventors: Insoo Kim (Seongnam-si, KR); Kikyung Kim (Hwaseong-si, KR); Seungju Han (Seoul, KR); Jiwon Baek (Hwaseong-si, KR); Jaejoon Han (Seoul, KR)

Assignee: Samsung Electronics Co., Ltd.

G06V10/806G06V10/7715G06V10/82

Abstract

Claims (70)

1 . A processor-implemented method, the method comprising:

extracting feature maps comprising local feature representations from an input image;

generating a global feature representation corresponding to the input image by fusing the local feature representations;

estimating a first recognition result corresponding to the local feature representations, using a first recognition model, and

estimating a second recognition result corresponding to the global feature representation, using a second recognition model, wherein the second recognition model comprises a classification model configured to estimate a classification result from the global feature representation; and

performing a recognition task on the input image based on the first recognition result corresponding to the local feature representations and the second recognition result corresponding to the global feature representation.

2 . The method of claim 1 , wherein the generating of the global feature representation comprises:

fusing pooling results corresponding to the local feature representations.

3 . The method of claim 2 , wherein the pooling comprises global average pooling.

4 . The method of claim 1 , wherein the generating of the global feature representation comprises:

performing an attention mechanism using query data pre-trained in association with the recognition task.

5 . The method of claim 4 , wherein the generating of the global feature representation by performing the attention mechanism comprises:

determining key data and value data corresponding to the local feature representations;

determining a weighted sum of the value data based on similarity between the key data and the query data; and

determining the global feature representation based on the weighted sum.

6 . The method of claim 1 , wherein the first recognition model comprises:

an object detection model configured to estimate a detection result from the local feature representations.

7 . The method of claim 6 , wherein the detection result comprises:

one or more of bounding box information, objectness information, or class information, and

wherein the classification result comprises:

one or more of multi-class classification information, context classification information, or object count information.

8 . The method of claim 6 , further comprising training the first model and training the second model, wherein the training the first recognition model affects the training of the second recognition model, and the training of the second recognition model affects the training of the first recognition model.

9 . The method of claim 8 , wherein the training of the first model and the training of the second model comprises:

using an in-training feature extraction model, or a trained feature extraction model, to extract training feature maps comprising in-training local feature representations from a training input image;

using an in-training feature fusion model, or a trained fusion model, to determine a training global feature representation corresponding to the training input image by fusing the training local feature representations;

using an in-training first recognition model to estimate a training first recognition result corresponding to the training local feature representations;

using an in-training second recognition model to estimate a training second recognition result corresponding to the training global feature representation; and

generating the first model and the second model by training the in-training first recognition model and the in-training second recognition model together based on the training first recognition result and the training second recognition result.

10 . The method of claim 6 , wherein the performing of the recognition task further comprises:

determining a task result recognized by the recognition task by fusing the first recognition result and the second recognition result.

11 . The method of claim 1 , wherein the recognition task corresponds to:

one of plural task candidates, the task candidates having respectively associated pre-trained query data items, and

wherein the generating of the global feature representation comprises:

selecting, from among the pre-trained query data items, a query data item associated with the recognition task; and

determining the global feature representation by performing an attention mechanism based on the selected query data item.

12 . The method of claim 1 , further comprising capturing the input image using a camera.

13 . A processor-implemented method, the method comprising:

using a feature extraction model to extract feature maps comprising local feature representations from an input image;

using a feature fusion model to determine a global feature representation corresponding to the input image by fusing the local feature representations;

using a first recognition model to estimate a first recognition result corresponding to the local feature representations;

using a second recognition model to estimate a second recognition result corresponding to the global feature representation, wherein the second recognition model comprises a plurality of classification models respectively corresponding to task candidates; and

based on the first recognition result corresponding to the local feature representations and the second recognition result corresponding to the global feature representation, training one or more of the feature extraction model, the feature fusion model, the first recognition model, or the second recognition model.

14 . The method of claim 13 , further comprising determining a training loss based on the first recognition result and the second recognition result, wherein the training is based on the training loss.

15 . The training method of claim 13 , wherein the first recognition model and the second recognition model are trained as an integrated model to influence the training of both the first recognition model and the second recognition model.

16 . The method of claim 13 , wherein the feature fusion model is configured to:

determine the global feature representation by performing an attention mechanism based on query data corresponding to a current task candidate among the task candidates, and

wherein the determining of the training loss comprises:

determining the training loss by applying a classification result of a classification model corresponding to the current task candidate among the task candidates as the second recognition result.

17 . The method of claim 13 , wherein training the first recognition model affects training the second recognition model and training the second recognition model affects training the first recognition model.

18 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the method of claim 1 .

19 . An electronic apparatus, comprising:

one or more processors comprising processing circuitry; and

a memory comprising one or more storage media storing instructions that, when executed individually or collectively by the one or more processors, cause the electronic device to: extract feature maps comprising respective local feature representations from an input image;

determine a global feature representation corresponding to the input image by fusing the local feature representations;

use a first recognition model to estimate a first recognition result corresponding to the local feature representations;

use a second recognition model to estimate a second recognition result corresponding to the global feature representation, wherein the second recognition model comprises a classification model configured to estimate a classification result from the global feature representation; and

perform a recognition task on the input image based on the first recognition result corresponding to the local feature representations and the second recognition result corresponding to the global feature representation.

20 . The electronic apparatus according to claim 19 , further comprising a camera configured to generate the input image.

21 . The electronic apparatus of claim 19 , wherein the processor is further configured to:

determine the global feature representation by fusing pooling results corresponding to the local feature representations.

22 . The electronic apparatus of claim 19 , wherein the processor is further configured to:

determine the global feature representation by performing an attention mechanism using query data pre-trained in response to the recognition task.

23 . The electronic apparatus of claim 22 , wherein the attention mechanism comprises a vision transformer model that performs the fusing based on similarity of keys and values of the query data.

24 . The electronic apparatus of claim 19 , wherein the processor is further configured to select between the first recognition model and the second recognition model based on the recognition task.

25 . The electronic apparatus of claim 19 , wherein the first recognition model comprises:

an object detection model configured to estimate a detection result corresponding to each of the local feature representations, and

wherein the second recognition model comprises:

a classification model configured to estimate a classification result corresponding to the global feature representation.

26 . The electronic apparatus of claim 25 , wherein the processor is further configured to:

determine a task result of the recognition task by fusing the first recognition result and the second recognition result.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (48)