Granted Patent B2

US 12,363,249 · App. 18/450,630 · Granted Jul 15, 2025

Inventors: Mahesh P J (Bengaluru, IN); Pavan Sudheendra (Bengaluru, IN); Narasimha Gopalakrishna Pai (Bengaluru, IN); Prityush Chandra (Bengaluru, IN); Chevuru Sai Jaswanth (Bengaluru, IN)

Assignee: Samsung Electronics Co., Ltd.

H04N5/2621G06T7/10G06T9/00H04N23/959G06T2207/10024G06T2207/20084

Abstract

Claims (67)

1. A method for generation of a plurality of portrait effects in an electronic device, the method comprising:

feeding an image captured from the electronic device into an encoder pre-learned using a plurality of features corresponding to the plurality of portrait effects;

extracting, using the encoder, at least one of one or more low level features and one or more high level features from the image;

generating, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more first decoders; and

generating, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features by passing the image through one or more second decoders, wherein each of the one or more first portrait effect, and the one or more second portrait effects is generated in a single inference.

2. The method of claim 1 ,

wherein the one or more first decoders comprises a Bokeh decoder, the one or more second decoders comprises a High Key decoder, and a Low Key decoder, and

wherein the one or more first portrait effects is at least one of a Big circle effect, a studio effect or a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait effect, a color backdrop effect, a color point effect, a spin effect, or a zoom effect.

3. The method of claim 1 ,

wherein the one or more first portrait effects relates to depth-related camera features, and

wherein the one or more second portrait effects relates to segmentation-related camera features.

4. The method of claim 1 , wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single deep neural network (DNN) model.

5. The method of claim 4 , further comprising:

training the single DNN model, wherein training the single DNN model comprises:

generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules,

training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image, and

training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.

6. The method of claim 5 , wherein the generating of the ground truth data for the one or more first portrait effects comprises:

generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and

generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.

7. The method of claim 5 , wherein the generating of the ground truth data for the one or more second portrait effects comprises:

generating a segmentation mask of an input color image;

generating a high-resolution matting mask from the segmentation mask;

refining the generated matting mask; and

generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.

8. The method of claim 5 , wherein the training of the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder comprises:

training the encoder, the one or more first decoders, and the one or more second decoders in a first stage of the plurality of stages to provide an input image as an output image;

training the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effects using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and

training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.

9. A system for generation of a plurality of portrait effects in an electronic device, the system comprising:

an encoder configured to:

receive an image captured from the electronic device, wherein the encoder is pre-learned using a plurality of features corresponding to the plurality of portrait effects, and

extract at least one of one or more low level features and one or more high level features from the image;

one or more first decoders to generate, for the image, one or more first portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features; and

one or more second decoders to generate, for the image, one or more second portrait effects of the plurality of portrait effects based on the at least one of the one or more high level features and the one or more low level features, wherein each of the one or more first, and the one or more second portrait effects is generated in a single inference.

10. The system of claim 9 ,

wherein the one or more first decoders comprises a Bokeh decoder, and the one or more second decoders comprise a High Key decoder, and a Low Key decoder, and

wherein the one or more first portrait effects is a Bokeh effect associated with the Bokeh decoder, the one or more second portrait effects is at least one of a High Key portrait effect, a Low Key portrait effect, a color backdrop effect, a color point effect, a spin effect, or a zoom effect.

11. The system of claim 9 , wherein the encoder, the one or more first decoders, and the one or more second decoders are comprised within a single DNN model.

12. The system of claim 11 , wherein the single DNN model is trained by:

generating ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects using a plurality of data modules;

training the encoder in a plurality of stages using the generated ground truth to extract the at least one of the one or more low level features and the one or more high level features from the image; and

training the encoder, the one or more first decoders, the one or more second decoders, and a defocus map decoder associated with the one or more first decoders in a plurality of stages using the generated ground truth data to generate the one or more first and the one or more second portrait effect.

13. The system of claim 12 , wherein the ground truth data for the one or more first portrait effects is generated by:

generating the ground truth data using a shallow depth of field of an input color image captured using a first aperture of a lens of a camera and a wide depth of field of the input color image captured using a second aperture of the lens of the camera; and

generating a ground truth defocus map by generating a depth map of the input color image and generating a defocus map based on the depth map to generate the ground truth data for the one or more first portrait effects.

14. The system of claim 12 , wherein the ground truth data for the one or more second portrait effects is generated by:

generating a segmentation mask of an input color image;

generating a high-resolution matting mask from the segmentation mask;

refining the generated matting mask; and

generating the one or more second portrait effects on the input color image by changing one or more color parameters of the input color image and replacing a background of the input color image with a predetermined background based on the one or more second portrait effects.

15. The system of claim 12 , wherein the encoder, the one or more first decoders, the one or more second decoders, and the defocus map decoder are trained by:

training the encoder, the one or more first decoders, and the one or more second decoders, in a first stage of the plurality of stages to provide an input image as an output image;

training the encoder, the one or more first decoders, and the one or more second decoders, and the defocus map decoder in a series of second stages of the plurality of stages to generate the one or more first portrait effect, and the one or more second portrait effect, using the ground truth data for each of the one or more first portrait effects, and the one or more second portrait effects and the input image; and

training the encoder, the one or more first decoders and the defocus map decoder to generate the one or more first portrait effects using the ground truth data for the one or more first portrait effects and the input image.

16. The system of claim 9 ,

wherein the one or more first portrait effects relates to depth-related camera features, and

wherein the one or more second portrait effects relates to segmentation-related camera features.

17. The system of claim 9 , further comprising:

a memory;

a processor;

a communicator;

a display;

one or more cameras; and

an image processor,

wherein the memory stores the plurality of portrait effects and information related to the generation of the plurality of portrait effects, and

wherein the memory stores instructions to be executed by the processor for generating the plurality of portrait effects.

Assignments (1)

Priority Claims (2)

Continuity (2)

References Cited (30)