Granted Patent B2

US 12,470,664 · App. 18/202,283 · Granted Nov 11, 2025

Inventors: Hyugjae Lee (Gyeonggi-do, KR); Byumhyuk Koo (Gyeonggi-do, KR); Rokkyu Lee (Gyeonggi-do, KR); Gunhan Park (Gyeonggi-do, KR)

Assignee: NHN Corporation

H04N5/272G06Q30/0643G06T19/006G06V40/103H04N2005/2726

Abstract

Claims (52)

1 . A computer device for providing a virtual try-on image, the computer device comprising:

a camera interface connected to a camera;

a display interface connected to a display device;

a storage medium; and

a processor configured to:

communicate with the camera and the display device through the camera interface and the display interface, respectively;

receive input images, including a user object and generated by the camera, through the camera interface;

generate pose estimation data representing a pose of the user object included in one of the received input images, wherein the pose estimation data comprises first keypoints representing body parts of the user object;

select, among the received input images, an input image having the user object of which pose represented by the pose estimation data matches a reference pose, wherein the storage medium is configured to store second keypoints corresponding to the reference pose;

determine whether the pose estimation data matches the reference pose by determining whether the first keypoints of the pose estimation data match the second keypoints corresponding to the reference pose;

generate the virtual try-on image by synthesizing a clothes object with the user object included in the selected input image; and

control the display device through the display interface to output the virtual try-on image through the display interface.

2 . The computer device of claim 1 , wherein a neural network is trained to determine whether keypoints of a first pose and keypoints of a second pose match each other by inputting the keypoints of the first pose and the keypoints of the second pose to the neural network, and

wherein the processor is configured to determine whether the first keypoints of the pose estimation data match the second keypoints corresponding to the reference pose based on data which is output from the neural network by inputting the first keypoints of the pose estimation data and the second keypoints corresponding to the reference pose to the neural network.

3 . The computer device of claim 1 , wherein the processor is configured to generate the virtual try-on image by performing image harmonization on the user object included in the selected input image and the clothes object overlapping the user object included in the selected input image.

4 . The computer device of claim 1 , wherein the processor is configured to:

generate a first synthesized image by synthesizing the clothes object with the user object included in the selected input image; and

generate a second synthesized image to be the virtual try-on image by synthesizing the first synthesized image with a background image which is to be overlapped with the first synthesized image.

5 . The computer device of claim 4 , wherein:

the first synthesized image is generated by performing image harmonization on the user object included in the selected input image and the clothes object overlapping the user object included in the selected input image, and

the second synthesized image is generated by performing the image harmonization on the background image and the first synthesized image overlapping the background image.

6 . The computer device of claim 1 , further comprising a communicator connected to a network,

wherein the processor is configured to receive the clothes object from a client server through the communicator.

7 . A computerized method for providing a virtual try-on image, the computerized method comprising:

generating input images including a user object by photographing a user using a camera;

generating pose estimation data representing a pose of the user object included in one of the generated input images, wherein the pose estimation data comprises first keypoints representing body parts of the user object;

selecting, among the generated input images, an input image having the user object of which pose represented by the pose estimation data matches a reference pose;

determining whether the pose estimation data matches the reference pose by determining whether the first keypoints of the pose estimation data match second keypoints corresponding to the reference pose;

generating the virtual try-on image by synthesizing a clothes object with the user object included in the selected input image; and

outputting the virtual try-on image using a display device.

8 . The computerized method of claim 7 , wherein the generating of the virtual try-on image comprises:

generating a first synthesized image by synthesizing the clothes object with the user object included in the selected input image; and

generating a second synthesized image to be the virtual try-on image by synthesizing the first synthesized image with a background image which is to be overlapped with the first synthesized image.

9 . A computer device for providing user experience by visualizing background images, the computer device comprising:

a camera interface connected to a camera;

a display interface connected to a display device;

a storage medium; and

a processor configured to:

communicate with the camera and the display device through the camera interface and the display interface, respectively;

receive input images, generated by the camera photographing a user, through the camera interface;

generate pose estimation data associated with an user object obtained from one of the input images, wherein the pose estimation data comprises first keypoints representing body parts of the user object;

select, among the received input images, an input image having the user object of which pose represented by the pose estimation data matches a reference pose, wherein the storage medium is configured to store second keypoints corresponding to the reference pose;

determine the one of the input images, from which the user object is obtained, as the selected input image when a pose of the user object associated with the pose estimation data matches a reference pose;

generate a first synthesized image by performing first image harmonization on the user object included in the input image selected among the input images and a clothes object overlapping the user object;

generate a second synthesized image by performing second image harmonization on one background image among the background images and the first synthesized image overlapping the one background image; and

output the second synthesized image by controlling the display device through the display interface.

10 . The computer device of claim 9 , wherein the processor is configured to convert the clothes object in association with the user object by processing the user object and the clothes object through a first convolutional neural network trained to perform the first image harmonization,

wherein the first convolutional neural network comprises at least one first convolutional encoder layer and at least one first convolutional decoder layer, and

wherein the first synthesized image comprises at least a part of the user object and the converted clothes object overlapping the user object.

11 . The computer device of claim 10 , wherein the processor is configured to convert the first synthesized image in association with the background image by processing the background image and the first synthesized image through a second convolutional neural network trained to perform the second image harmonization,

wherein the second convolutional neural network comprises at least one second convolutional encoder layer and at least one second convolutional decoder layer, and

wherein the second synthesized image comprises at least a part of the background image and the converted first synthesized image overlapping the background image.

Assignments (2)

Priority Claims (1)

Continuity (1)

References Cited (26)