Granted Patent B2

US 12,573,191 · App. 18/136,614 · Granted Mar 10, 2026

Inventors: Soomin Kang (Suwon-si, KR); Youngchan Song (Suwon-si, KR); Tammy Lee (Suwon-si, KR); Jaeyeon Park (Suwon-si, KR); Hanul Shin (Suwon-si, KR); Iljun Ahn (Suwon-si, KR)

Assignee: SAMSUNG ELECTRONICS CO., LTD.

G06V10/82G06F16/532G06T5/50G06T5/70G06V10/7715G06V10/774G06T2207/20021G06T2207/20081G06T2207/20084G06T2207/20192

Abstract

Claims (60)

1 . An image processing apparatus for processing an image by using at least one neural network, the image processing apparatus comprising:

a memory storing at least one instruction; and

at least one processor configured to execute the at least one instruction to:

obtain first feature data, based on a first image,

divide the first feature data into a plurality of first patches, each of the plurality of first patches comprising M pixels,

obtain pieces of second feature data corresponding to each of pixels included in each of the plurality of first patches by performing a first self-attention operation on the pixels included in the same patch for each of the plurality of first patches,

obtain third feature data, based on the first image,

divide the pieces of second feature data into a plurality of second patches, each of the plurality of second patches comprising M pixels,

obtain pieces of fourth feature data corresponding to each of the plurality of second patches, by performing a second self-attention operation on the plurality of second patches, and

generate a second image, based on the pieces of second feature data and the pieces of fourth feature data.

2 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:

obtain the pieces of second feature data respectively corresponding to the plurality of first patches, based on information about peripheral areas of each of the plurality of first patches, and

obtain the pieces of fourth feature data respectively corresponding to the plurality of second patches, based on information about peripheral areas of each of the plurality of second patches.

3 . The image processing apparatus of claim 1 , wherein each of the plurality of first patches comprises one pixel.

4 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:

obtain pieces of query data, pieces of key data, and pieces of value data based on the first feature data, the pieces of query data, the pieces of key data, and the pieces of value data respectively corresponding to the plurality of first patches,

obtain a weight matrix, based on the pieces of query data and the pieces of key data, and obtain the pieces of second feature data, based on the pieces of value data and the weight matrix.

5 . The image processing apparatus of claim 4 , wherein the at least one processor is further configured to execute the at least one instruction to:

obtain a correlation matrix, based on the pieces of query data and the pieces of key data, and obtain the weight matrix by applying, to the correlation matrix, a position bias based on a size of the first image and sizes of images used to train the at least one neural network.

6 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:

transform the third feature data divided into third areas comprising a first number of pixels to be divided into the plurality of second patches, and obtain the pieces of fourth feature data by performing the second image processing on each of the plurality of second patches.

7 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:

obtain pieces of first query data, pieces of first key data, and pieces of first value data based on the third feature data, the pieces of first query data, the pieces of first key data, and the pieces of first value data respectively corresponding to third areas comprising a first number of pixels,

obtain pieces of second query data, pieces of second key data, and pieces of second value data corresponding to the plurality of second patches, by grouping the pieces of first query data, the pieces of first key data, and the pieces of first value data to respectively correspond to the plurality of second patches,

obtain a weight matrix, based on the pieces of second query data and the pieces of second key data, and

obtain the pieces of fourth feature data, based on the pieces of second value data and the weight matrix.

8 . The image processing apparatus of claim 1 , wherein the third feature data is obtained from the pieces of second feature data.

9 . The image processing apparatus of claim 1 , wherein the at least one neural network comprises at least one convolutional neural network, and

wherein the at least one processor is further configured to execute the at least one instruction to extract the first feature data from the first image by using the at least one convolutional neural network.

10 . The image processing apparatus of claim 1 , wherein the at least one neural network comprises at least one convolutional neural network, and

wherein the at least one processor is further configured to execute the at least one instruction to:

obtain fifth feature data, based on the pieces of second feature data and the pieces of fourth feature data, and

obtain the second image from the fifth feature data, by using the at least one convolutional neural network.

11 . An operating method of an image processing apparatus for processing an image by using at least one neural network, the operating method comprising:

obtaining first feature data, based on a first image;

dividing the first feature data into a plurality of first patches, each of the plurality of first patches comprising M pixels;

obtaining pieces of second feature data corresponding to each of pixels included in each of the plurality of first patches, by performing a first self-attention operation on the pixels included in the same patch for each of the plurality of first patches;

obtaining third feature data, based on the first image;

dividing the pieces of second feature data into a plurality of second patches, each of the plurality of second patches comprising M pixels;

obtaining pieces of fourth feature data corresponding to each of the plurality of second patches by performing a second self-attention operation on the plurality of second patches; and

generating a second image, based on the pieces of second feature data and the pieces of fourth feature data.

12 . The operating method of claim 11 , wherein the obtaining the pieces of second feature data comprises obtaining the pieces of second feature data respectively corresponding to the plurality of first patches, based on information about peripheral areas of each of the plurality of first patches, and

wherein the obtaining the pieces of fourth feature data comprises obtaining the pieces of fourth feature data respectively corresponding to the plurality of second patches, based on information about peripheral areas of each of the plurality of second patches.

13 . The operating method of claim 11 , wherein each of the plurality of first patches comprises one pixel.

14 . The operating method of claim 11 , wherein the obtaining of the pieces of second feature data comprises:

obtaining pieces of query data, pieces of key data, and pieces of value data based on the first feature data, the pieces of query data, the pieces of key data, and the pieces of value data respectively corresponding to the plurality of first patches;

obtaining a weight matrix, based on the pieces of query data and the pieces of key data; and

obtaining the pieces of second feature data, based on the pieces of value data and the weight matrix.

15 . The operating method of claim 14 , wherein the obtaining the weight matrix comprises:

obtaining a correlation matrix, based on the pieces of query data and the pieces of key data, and

obtaining the weight matrix by applying, to the correlation matrix, a position bias based on a size of the first image and sizes of images used to train the at least one neural network.

16 . The operating method of claim 11 , wherein the obtaining the pieces of fourth feature data comprises:

transforming the third feature data divided into third areas comprising a first number of pixels to be divided into the plurality of second patches; and

obtaining the pieces of fourth feature data by performing the second image processing on each of the plurality of second patches.

17 . The operating method of claim 11 , wherein the obtaining the pieces of fourth feature data comprises:

obtaining pieces of first query data, pieces of first key data, and pieces of first value data based on the third feature data, the pieces of first query data, the pieces of first key data, and the pieces of first value data respectively corresponding to third areas comprising a first number of pixels;

obtaining pieces of second query data, pieces of second key data, and pieces of second value data corresponding to the plurality of second patches, by grouping the pieces of first query data, the pieces of first key data, and the pieces of first value data to respectively correspond to the plurality of second patches;

obtaining a weight matrix, based on the pieces of second query data and the pieces of second key data; and

obtaining the pieces of fourth feature data, based on the pieces of second value data and the weight matrix.

18 . At least one non-transitory computer-readable recording medium storing a program for executing that is executed by a processor to perform the operating method of claim 11 .

Assignments (1)

Priority Claims (2)

Continuity (2)

References Cited (56)