Method and apparatus for aligning 3D model

1. A method of aligning a three-dimensional (3D) model, the method comprising:

acquiring, by a processor, a first two-dimensional (2D) image including an object of a first pose;

generating a second 2D input image including the object of a second pose that is different from the first pose;

detecting, by the processor, a feature point of the object in the first 2D input image and the generated second 2D input image, using a neural network;

estimating, by the processor, a 3D pose of the object in the first 2D input image and the generated second 2D input image, using the neural network;

retrieving, by the processor, a target 3D model based on the estimated 3D pose; and

aligning, by the processor, the target 3D model and the object based on the feature point.

2. The method of claim 1 , wherein the acquiring of the at least one 2D input image comprises receiving a third 2D input image including the object of a third pose that is different from the first pose and/or the second pose.

3. The method of claim 1 , further comprising:

detecting the object in the first 2D input image.

4. The method of claim 1 , wherein the estimating of the 3D pose comprises:

classifying a type of the object using the neural network; and

estimating the 3D pose of the object based on a result of the classification using the neural network.

5. The method of claim 1 , wherein the retrieving of the target 3D model comprises:

acquiring a first feature of the object in the first 2D input image;

acquiring a second feature of a candidate 3D model from among candidate 3D models; and

determining the candidate 3D model to be the target 3D model based on the first feature and the second feature.

6. The method of claim 5 , wherein the determining comprises:

calculating a similarity between the first feature and the second feature; and

determining the candidate 3D model to be the target 3D model based on the similarity.

7. The method of claim 1 , further comprising:

adjusting the object or the target 3D model based on the estimated 3D pose, the feature point of the object, and a feature point of the target 3D model.

8. The method of claim 7 , wherein the adjusting comprises:

adjusting the target 3D model or the object using the estimated 3D pose; and

readjusting the adjusted object or the adjusted target 3D model based on the feature point of the object and the feature point of the target 3D model.

9. The method of claim 7 , wherein the adjusting comprises:

adjusting the object or the target 3D model based on the feature point of the object and the feature point of the target 3D model; and

readjusting the adjusted object or the adjusted target 3D model based on the estimated 3D pose.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

11. A method of training a neural network, the method comprising:

acquiring, by a processor, a first training two-dimensional (2D) input image including an object of a first pose;

generating a second training 2D input image including the object of a second pose that is different from the first pose;

estimating, by the processor, a three-dimensional (3D) pose of the object in the first training 2D input image and the second training 2D input image, using the neural network;

retrieving, by the processor, a target 3D model based on the estimated 3D pose;

detecting, by the processor, a feature point of the object in the first training 2D input image and the second training 2D input image using the neural network; and

training, by the processor, the neural network based on the estimated 3D pose or the detected feature point.

12. The method of claim 11 , wherein the estimating of the 3D pose comprises:

classifying a type of the object using the neural network;

estimating the 3D pose of the object based on a result of the classification using the neural network, and

the training of the neural network comprises training the neural network based on the classified type.

13. The method of claim 11 , further comprising:

acquiring a composite image of at least one candidate 3D model of the estimated 3D pose; and

classifying a domain of each of the first training 2D input image and the composite image using the neural network,

wherein the training of the neural network comprises training the neural network based on the classified domain.

14. The method of claim 13 , wherein the acquiring of the composite image comprises acquiring a first composite image of a first candidate 3D model of the estimated 3D pose, a second composite image of the first candidate 3D model of a second pose, a third composite image of a second candidate 3D model of the estimated 3D pose, a fourth composite image of the second candidate 3D model of the second pose, and the at least one candidate 3D model comprising the first candidate 3D model and the second candidate 3D model.

15. The method of claim 14 , wherein a similarity between the first candidate 3D model and the object is greater than or equal to a threshold and a similarity between the second candidate 3D model and the object is less than the threshold.

16. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 12 .

17. An apparatus for aligning a three-dimensional (3D) model, the apparatus comprising:

a memory configured to store a neural network and instructions,

a processor configured to execute the instructions to

acquire a first two-dimensional (2D) image including an object of a first pose,

generate a second 2D input image including the object of a second pose that is different from the first pose,

detect a feature point of the object in the first 2D image and the generated second 2D input image, using the neural network,

estimate a 3D pose of the object in the first 2D image and the generated second 2D input image, using the neural network,

retrieve a target 3D model based on the estimated 3D pose, and

align the target 3D model and the object based on the feature point.