System and method for processing training dataset associated with synthetic image

1 . A training dataset generating system comprising:

a communication interface to receive a two-dimensional (2D) image obtained by photographing a target object; and

a controller circuit configured to generate, based on the 2D image and based on three-dimensional (3D) data for the target object, a training dataset comprising a synthetic image and comprising labeling information,

wherein the controller circuit is configured to generate the training data set by:

generating, based on the 3D data, a rendered image;

generating the synthetic image by synthesizing a style of the 2D image and a content of the rendered image, through a trained deep learning model;

extracting, based on at least one of the 3D data or the rendered image, the labeling information for the target object; and

generating the training dataset, and

wherein the controller circuit is configured to:

extract, via an encoder, one or more first features from the 2D image and one or more second features from the rendered image;

generate a synthetic feature by synthesizing the one or more first features and the one or more second features; and

generate, via a decoder, the synthetic image, based on the synthetic feature, through the trained deep learning model,

wherein the decoder comprises a plurality of deconvolutional layers for deconvolution of the synthetic feature, and is configured to determine weights of a plurality of kernels corresponding to the plurality of deconvolutional layers through the trained deep learning model, and

wherein the decoder is configured to generate, based on the weights of the plurality of kernels and based on the synthetic feature, the synthetic image.

2 . The training dataset generating system of claim 1 , wherein

the controller circuit is configured to:

convert, via a preprocessor, the 2D image to a first sketch image;

generate a generator image of a generator, based on the first sketch image, through the trained deep learning model; and

determine similarity between the generator image and the 2D image, to determine whether the similarity is greater than or equal to a threshold, and to send, to the image generator as a feedback, a determined result associated with the similarity.

3 . The training dataset generating system of claim 2 , wherein

the generator comprises a plurality of convolutional layers to convert the first sketch image to the generator image, and determines, based on the feedback, weights of a plurality of kernels corresponding to the plurality of convolutional layers.

4 . The training dataset generating system of claim 3 , wherein

the preprocessor converts the rendered image to a second sketch image, and

the generator generates, based on the weights of the plurality of kernels and based on the second sketch image, the synthetic image.

5 . The training dataset generating system of claim 1 , wherein

the controller circuit is configured to extract, from the 3D data, classification labeling information for the target object.

6 . The training dataset generating system of claim 1 , wherein

the 2D image comprises a color photo obtained by photographing a region comprising an assembly in which the target object is assembled with another object, wherein the style of the 2D image comprises texture and color information of the target object in the 2D image,

the 3D data comprises assembly data indicating the assembly and single data indicating the target object, and

the controller circuit is configured to extract shape labeling information for the target object and position labeling information for the target object from an object-rendered image generated based on the assembly data and the single data.

7 . The training dataset generating system of claim 1 , wherein

the 2D image comprises a color photo obtained by photographing a region comprising the target object not assembled with another object, and

the controller circuit is configured to extract shape labeling information for the target object and position labeling information for the target object from contour information for the target object shown in the rendered image.

8 . A training dataset generating method comprising:

generating, based on a two-dimensional (2D) image obtained by photographing a target object and based on three-dimensional (3D) data for the target object, a training dataset, wherein the generating comprises:

generating, by a controller circuit, a rendered image by rendering the 3D data;

generating, by the controller circuit, a synthetic image by synthesizing a style of the 2D image and a content of the rendered image together through a trained deep learning model;

extracting, by the controller circuit and from the 3D data or from the rendered image, labeling information for the target object; and

generating, by the controller circuit and based on the synthetic image and the labeling information for the target object, the training dataset,

wherein the generating of the synthetic image comprises:

extracting, by the controller circuit and via an encoder, a style feature from the 2D image and a content feature from the rendered image;

generating, by the controller circuit, a synthetic feature by synthesizing the style feature and the content feature together; and

generating, via a decoder, the synthetic image, based on the synthetic feature, through the trained deep learning model,

wherein the decoder comprises a plurality of deconvolutional layers for deconvolution of the synthetic feature, and determines weights of a plurality of kernels corresponding to the plurality of deconvolutional layers through the trained deep learning model, and

wherein the decoder generates, based on the weights of the plurality of kernels and based on the synthetic feature, the synthetic image.

9 . The training dataset generating method of claim 8 , wherein the generating of the synthetic image comprises:

training, by the controller circuit and based on the synthetic feature, a method for generating the synthetic image.

10 . The training dataset generating method of claim 9 , wherein

the training of the method for generating the synthetic image comprises:

determining weights of a plurality of kernels corresponding to a plurality of deconvolutional layers for deconvolution of the synthetic feature; and

generating, based on the synthetic feature and based on the weights of the plurality of kernels, the synthetic image.

11 . The training dataset generating method of claim 9 , wherein

the generating of the synthetic image further comprises

generating, based on the synthetic feature and based on weights of a plurality of kernels determined through the training, a synthetic image included in the training dataset.

12 . The training dataset generating method of claim 8 , wherein

the generating of the synthetic image comprises:

generating, by the controller circuit, a first sketch image by converting, based on a sketch style, the 2D image;

determining, by the controller circuit, weights of a plurality of kernels corresponding to a plurality of deconvolutional layers by training a method for converting the first sketch image to the 2D image through training;

generating, by the controller circuit, a second sketch image by converting, based on the sketch style, the rendered image; and

generating, by the controller circuit and based on the second sketch image and the weights of the plurality of kernels, the synthetic image.

13 . The training dataset generating method of claim 12 , wherein

the determining of the weights of the plurality of kernels comprises:

generating, based on the first sketch image, a generator image;

determining similarity between the generator image and the 2D image;

determining whether the similarity is greater than or equal to a threshold; and

determining, based on a determined result associated with the similarity, the weights of the plurality of kernels.

14 . The training dataset generating method of claim 12 , wherein

the generating of the synthetic image further comprises

generating, based on the second sketch image and based on the weights of the plurality of kernels determined through the training, a synthetic image included in the training dataset.

15 . The training dataset generating method of claim 8 , wherein

the extracting of the labeling information for the target object comprises

extracting classification labeling information for the target object from the 3D data.

16 . The training dataset generating method of claim 8 , wherein

the 2D image comprises a color photo obtained by photographing a region comprising an assembly in which the target object is assembled with another object,

the 3D data comprises assembly data indicating the assembly and single data indicating the target object, and

the extracting of the labeling information for the target object comprises:

generating, by the controller circuit, target removal data by removing the target object from the assembly;

generating, by the controller circuit, black data by converting a color indicated by the target removal data to a black color;

generating, by the controller circuit, 3D object data by combining the black data with the single data; and

generating, by the controller circuit, an object-rendered image by rendering the 3D object data.

17 . The training dataset generating method of claim 16 , wherein

the extracting of the labeling information for the target object further comprises

extracting, by the controller circuit and from the object-rendered image, shape labeling information corresponding to contour information for the target object.

18 . The training dataset generating method of claim 17 , wherein

the extracting of the labeling information for the target object further comprises

extracting, by the controller circuit, position labeling information corresponding to information indicating a horizontal upper limit of the contour information, a horizontal lower limit of the contour information, a vertical upper limit of the contour information, and a vertical lower limit of the contour information.

19 . The training dataset generating method of claim 8 , wherein

the 2D image comprises a color photo obtained by photographing a region comprising the target object not assembled with another object, and

the extracting of the labeling information for the target object comprises

extracting, by the controller circuit, shape labeling information corresponding to contour information for the target object from the rendered image.

20 . The training dataset generating method of claim 19 , wherein

the extracting of the labeling information for the target object further comprises

extracting, by the controller circuit and from the contour information for the target object, the shape labeling information for the target object and position labeling information for the target object.

21 . A non-transitory computer-readable medium storing instructions that cause:

generating, based on a two-dimensional (2D) image obtained by photographing a target object and based on three-dimensional (3D) data for the target object, a training dataset, wherein the generating comprises:

generating, by a controller circuit, a rendered image by rendering the 3D data;

generating, by the controller circuit, a synthetic image by synthesizing a content of the 2D image and a style of the rendered image together through a trained deep learning model;

extracting, by the controller circuit and from the 3D data or from the rendered image, labeling information for the target object; and

generating, by the controller circuit and based on the synthetic image and the labeling information for the target object, the training dataset,

wherein the generating of the synthetic image comprises:

extracting, via an encoder, a style feature from the 2D image and a content feature from the rendered image;

generating a synthetic feature by synthesizing the style feature and the content feature together; and

generating, via a decoder, the synthetic image, based on the synthetic feature, through the trained deep learning model,

wherein the decoder comprises a plurality of deconvolutional layers for deconvolution of the synthetic feature, and determines weights of a plurality of kernels corresponding to the plurality of deconvolutional layers through the trained deep learning model, and

wherein the decoder generates, based on the weights of the plurality of kernels and based on the synthetic feature, the synthetic image.

22 . The non-transitory computer-readable medium of claim 21 , wherein

the generating of the synthetic image comprises:

generating a first sketch image by converting, based on a sketch style, the 2D image;

determining weights of a plurality of kernels corresponding to a plurality of deconvolutional layers by training a method for converting the first sketch image to the 2D image through training;

generating a second sketch image by converting, based on the sketch style, the rendered image; and

generating, based on the second sketch image and the weights of the plurality of kernels, the synthetic image.

23 . The non-transitory computer-readable medium of claim 22 , wherein the determining of the weights of the plurality of kernels comprises:

generating, based on the first sketch image, a generator image;

determining similarity between the generator image and the 2D image;

determining whether the similarity is greater than or equal to a threshold; and

determining, based on a determined result associated with the similarity, the weights of the plurality of kernels.

24 . The non-transitory computer-readable medium of claim 21 , wherein

the 2D image comprises a color photo obtained by photographing a region comprising an assembly in which the target object is assembled with another object,

the 3D data comprises assembly data indicating the assembly and single data indicating the target object, and

the extracting of the labeling information for the target object comprises:

generating target removal data by removing the target object from the assembly;

generating black data by converting a color indicated by the target removal data to a black color;

generating 3D object data by combining the black data with the single data; and

generating an object-rendered image by rendering the 3D object data.