Granted Patent B2

US 12,684,237 · App. 17/982,029 · Granted Jul 14, 2026

Inventors: Kinam Kwon (Suwon-si, KR); Heewon Kim (Seoul, KR); Kyoung Mu Lee (Seoul, KR); Hyong Euk Lee (Suwon-si, KR)

Assignees: Samsung Electronics Co., Ltd.; Seoul National University R&DB Foundation

H04N23/682G06T5/50G06T5/60G06T5/70G06T5/73G06T5/77G06T5/92G06T2200/24G06T2207/20081G06T2207/20084

Abstract

Claims (93)

1 . A processor-implemented method with image processing, the method comprising:

providing retouch result candidates of an input image to a user in response to applying vector value candidates to a single style vector comprising m-dimensions of style, where m is at least two, wherein the style vector specifies a vector value of each dimension of m-dimensions;

determining vector values of the style vector by adjusting the m-dimensions of the style vector based on user selections of the retouch result candidates;

generating an adjustment parameter set corresponding to the determined vector values of the style vector; and

generating a retouch result by adjusting the input image based on the adjustment parameter set,

wherein the providing of the retouch result candidates of the input image to the user and the determining of the vector values of the style vector is based on the selection of the user comprises:

providing first retouch result candidates to the user in response to changing vector value candidates of an m1-dimension of the m-dimensions;

determining a vector value of the m1-dimension of the style vector based on a first selection of the user for the first retouch result candidates;

providing second retouch result candidates to the user in response to changing vector value candidates of an m2-dimension of the m-dimensions; and

determining a vector value of the m2-dimension of the style vector based on a second selection of the user for the second retouch result candidates,

wherein, in the providing of the first retouch result candidates to the user in response to changing the vector value candidates of the m1-dimension, vector value candidates of remaining dimensions other than the m1-dimension in the m-dimensions are fixed, and

wherein, in the providing of the second retouch result candidates to the user in response to changing the vector value candidates of the m2-dimension, the determined vector value of the m1-dimension are fixed.

2 . The method of claim 1 , wherein the m-dimensions is three-dimensions.

3 . The method of claim 1 , wherein, in the providing of the first retouch result candidates to the user and the providing of the second retouch result candidates to the user, the vector value candidates of the m1-dimension and the vector value candidates of the m2-dimension are changed under a control of the user.

4 . The method of claim 1 , wherein the providing of the first retouch result candidates to the user comprises:

determining a candidate adjustment parameter set corresponding to a vector value candidate of the m1-dimension based on the changing; and

generating a candidate retouch result by adjusting the input image based on the candidate adjustment parameter set.

5 . The method of claim 1 , wherein the determining of the adjustment parameter set comprises determining the adjustment parameter set using a decoding model based on deep learning.

6 . The method of claim 5 , wherein the decoding model is trained through an encoder-decoder framework.

7 . The method of claim 5 , wherein the decoding model is trained through operations of:

inputting a sample input image and a sample retouch image to an encoding model based on a neural network;

inputting an output of the encoding model corresponding to a sample style vector to the decoding model;

inputting an output of the decoding model corresponding to a sample adjustment parameter set to an image adjustment model;

acquiring an output of the image adjustment model corresponding to a sample retouch result; and

training the encoding model and the decoding model such that a difference between the sample retouch image and the sample retouch result is reduced.

8 . The method of claim 1 , wherein the adjustment parameter set comprises a parameter that adjusts any one or any combination of any two or more of a digital gain, a white balance, a color correction, a gamma correction, tone mapping, denoising, and deblurring.

9 . The method of claim 1 , wherein the generating of the retouch result comprises generating the retouch result by applying the adjustment parameter set to an image signal processing (ISP) pipeline set in advance.

10 . The method of claim 9 , wherein the ISP pipeline implements any one or any combination of any two or more of:

a first adjustment function to adjust a digital gain;

a second adjustment function to adjust a white balance;

a third adjustment function to perform a color correction;

a fourth adjustment function to perform a gamma correction;

a fifth adjustment function to perform tone mapping;

a sixth adjustment function to perform denoising; and

a seventh adjustment function to perform deblurring.

11 . The method of claim 10 , wherein the adjustment parameter set comprises an input value of any one or any combination of any two or more of the first adjustment function, the second adjustment function, the third adjustment function, the fourth adjustment function, the fifth adjustment function, the sixth adjustment function, and the seventh adjustment function.

12 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

13 . The method of claim 1 , wherein the determining of the vector values of the style vector comprises:

determining a vector value of a first dimension of the m-dimensions of the style vector;

setting the vector value of the first dimension; and

providing, while maintaining the set vector value of the first dimension, additional retouch result candidates by adjusting a vector value for a second dimension of the style vector.

14 . The method of claim 1 , wherein each user selection of a retouch result candidate determines a vector value for a respective dimension of the style vector, and

wherein new retouch result candidates are generated after each user selection using the determined vector value.

15 . An apparatus with image processing, the apparatus comprising:

one or more processors configured to:

provide retouch result candidates of an input image to a user in response to applying vector value candidates to a single style vector comprising m-dimensions of style, where m is at least two, wherein the style vector specifies a vector value of each dimension of m-dimensions;

determine vector values of the style vector by adjusting the m-dimensions of the style vector based on user selections of the retouch result candidates;

generating an adjustment parameter set corresponding to the determined vector values of the style vector; and

generate a retouch result by adjusting the input image based on the adjustment parameter set,

wherein, for the providing of the retouch result candidates and the determining of the vector values, the one or more processors are configured to:

provide first retouch result candidates to the user in response to changing vector value candidates of an m1-dimension of the m-dimensions;

determine a vector value of the m1-dimension of the style vector based on a first selection of the user for the first retouch result candidates;

provide second retouch result candidates to the user in response to changing vector value candidates of an m2-dimension of the m-dimensions; and

determine a vector value of the m2-dimension of the style vector based on a second selection of the user for the second retouch result candidates,

wherein, in the providing of the first retouch result candidates to the user in response to changing the vector value candidates of the m1-dimension, vector value candidates of remaining dimensions other than the m1-dimension in the m-dimensions are fixed, and

wherein, in the providing of the second retouch result candidates to the user in response to changing the vector value candidates of the m2-dimension, the determined vector value of the m1-dimension are fixed.

16 . The apparatus of claim 15 , wherein, in the providing of the first retouch result candidates to the user and the providing of the second retouch result candidates to the user, the vector value candidates of the m1-dimension and the vector value candidates of the m2-dimension are changed under a control of the user.

17 . The apparatus of claim 15 , further comprising a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the providing of the retouch result candidates, the determining of the vector value, the generating of the adjustment parameter set, and the generating of the retouch result.

18 . The apparatus of claim 15 , wherein each user selection of a retouch result candidate determines a vector value for a respective dimension of the style vector, and

wherein new retouch result candidates are generated after each user selection using the determined vector value.

19 . An electronic apparatus comprising:

a camera configured to generate an input image; and

one or more processors configured to:

provide retouch result candidates of the input image to a user in response to applying vector value candidates to a single style vector comprising m-dimensions of style, where m is at least two, wherein the style vector specifies a vector value of each dimension of m-dimensions;

determine vector values of the style vector by adjusting the m-dimensions of the style vector based on user selection of the retouch result candidates;

generate an adjustment parameter set corresponding to the determined vector values of the style vector; and

generate a retouch result by adjusting the input image based on the adjustment parameter set,

wherein, for the providing of the retouch result candidates and the determining of the vector values, the one or more processors are configured to:

provide first retouch result candidates to the user in response to changing vector value candidates of an m1-dimension of the m-dimensions;

determine a vector value of the m1-dimension of the style vector based on a first selection of the user for the first retouch result candidates;

provide second retouch result candidates to the user in response to changing vector value candidates of an m2-dimension of the m-dimensions; and

determine a vector value of the m2-dimension of the style vector based on a second selection of the user for the second retouch result candidates,

wherein, in the providing of the first retouch result candidates to the user in response to changing the vector value candidates of the m1-dimension, vector value candidates of remaining dimensions other than the m1-dimension in the m-dimensions are fixed, and

wherein, in the providing of the second retouch result candidates to the user in response to changing the vector value candidates of the m2-dimension, the determined vector value of the m1-dimension are fixed.

20 . The apparatus of claim 19 , wherein the determining of the vector values of the style vector comprises:

setting the vector value of the first dimension; and

providing, while maintaining the set vector value of the first dimension, additional retouch result candidates by adjusting a vector value for a second dimension of the style vector.

21 . The apparatus of claim 19 , wherein each user selection of a retouch result candidate determines a vector value for a respective dimension of the style vector, and

wherein new retouch result candidates are generated after each user selection using the determined vector value.

22 . A processor-implemented method with image processing, the method comprising:

generating retouch result candidates of an input image in response to applying vector value candidates to a single style vector comprising m-dimensions of style, where m is at least two, where the style vector specifies a vector value of each dimensions of m-dimensions;

determining vector values of the style vector based on the retouch result candidates;

determining an adjustment parameter set corresponding to the determined vector values of the style vector; and

generating a retouch result by adjusting the input image based on the adjustment parameter set,

wherein the generating of the retouch result candidates and the determining of the vector values comprises:

generating a first retouch result candidate in response to generating a vector value candidate of a first dimension; and

determining a vector value of the first dimension of the style vector based on the first retouch result candidate,

wherein, in the generating of the first retouch result candidates in response to generating the vector value candidate of the first dimension, vector value candidates of remaining dimensions other than the first dimension in the m-dimensions are fixed.

23 . The method of claim 22 , wherein the determining of the vector values of the style vector comprises:

setting the vector value of the first dimension; and

providing, while maintaining the set vector value of the first dimension, additional retouch result candidates by adjusting a vector value for a second dimension of the style vector.

24 . The method of claim 22 , wherein each user selection of a retouch result candidate determines a vector value for a respective dimension of the style vector, and

wherein new retouch result candidates are generated after each user selection using the determined vector value.

Assignments (2)

Priority Claims (2)

Continuity (1)

References Cited (53)