Method of training a neural network configured for converting 2D images into 3D models
A computer-implemented method of generating of a database for training a neural network configured for converting 2d images into 3d models comprising steps of: (a) obtaining 3d models; (b) rendering said 3d models in a 2d format from at least one view point; and (c) collecting pairs further comprising said rendered 2d image frame and said corresponding sampled 3d models each.
1. A computer-implemented method of generating a database for training a neural network, said neural network configured, when trained, for converting 2d images into 3d models; said database comprising a training collection of rendered 2d images and 3d models corresponding thereto; said method comprising steps of:
a. obtaining said 3d models by means of at least one of the following:
i. capturing a number of single volumetric frames of said at least one character in static poses and obtaining said 3d model from said number of single volumetric frames; and
ii. capturing a volumetric video of said at least one character being in motion and obtaining said 3d model from said volumetric video;
said at least one character at sub-steps i to iii being identical to each other or differ from each other;
b. generating said rendered 2d image by rendering said 3d model in a 2d image format from at least one view point; and
c. generating said database by collecting pairs, each of said pairs comprising said rendered 2d image and a corresponding sampled 3d model, said corresponding sampled 3d model comprising at least a portion of said 3d model generated at step (a);
d. said database training at least one neural network configured to convert at least one 2d image to a 3d model;
wherein said 3d model obtained at step (a) is an undivided 3d model, obtained directly, without a dummy model.
2. The method according to claim 1 , additionally comprising a step of sampling said 3d models obtained at step a.
3. The method according to claim 1 , wherein said rigging and skinning of said 3d model is performed by projecting said 3d model to at least one 2d image and calculating a 3d pose estimation based thereon.
4. The system according to claim 3 , wherein said rigging and skinning of said 3d model is performed in an automatic manner.
5. The method according to claim 1 , wherein at least one of said static poses is a T-pose or an A-pose.
6. A computer-implemented system for generating a database for training a neural network, said neural network configured, when trained, for converting 2d images into 3d models; said database comprising a collection of rendered 2d images and 3d models corresponding thereto; said computer-implemented system comprising:
a. a processor;
b. a memory storing instructions which, when executed by said processor, direct said processor to perform steps of:
i. obtaining said 3d models by means of at least one of the following:
1. capturing a number of single volumetric frames of said at least one character in static poses and obtaining said 3d model from said number of single volumetric frames; and
2. capturing a volumetric video of said at least one character being in motion and obtaining said 3d model from said volumetric video;
said at least one character at sub-steps 1 to 3 being identical to each other or differ from each other;
ii. generating said rendered 2d image by rendering said 3d model in a 2d image format from at least one view point; and
iii. generating said database by collecting pairs, each of said pairs comprising said rendered 2d image and a corresponding sampled 3d model, said corresponding sampled 3d model comprising at least a portion of said 3d model generated at step (a);
iv. said database training at least one neural network configured to convert at least one 2d image to a 3d model;
wherein said 3d model obtained at step (i) is an undivided 3d model, obtained directly, without a dummy model.
7. The system according to claim 6 , additionally comprising sampling said 3d models obtained at step i.
8. The system according to claim 6 , wherein said rigging and skinning of said 3d model is performed by projecting said 3d model to at least one 2d image and calculating a 3d pose estimation based thereon.
9. The system according to claim 8 , wherein said rigging and skinning of said 3d model is performed in an automatic manner.
10. The system according to claim 6 , wherein at least one of said static poses is a T-pose or an A-pose.
11. A non-transitory computer readable medium comprising instructions to a processor for performing a method of generating a training database for training a neural network, said neural network configured, when trained, for converting 2d images into 3d models; said instructions comprising steps of:
a. obtaining 3d models by means of at least one of the following:
i. capturing a number of volumetric frames of said at least one character in static poses and obtaining said 3d model from said number of volumetric frames; and
ii. capturing a volumetric video of said at least one character being in motion and obtaining said 3d model from said volumetric video;
said at least one character at sub-steps i to iii being identical to each other or differ from each other;
b. rendering said 3d models in a 2d image format from at least one view point; and
c. generating said database by collecting pairs, each of said pairs comprising said rendered 2d image and a corresponding sampled 3d model, said corresponding sampled 3d model comprising at least a portion of said 3d model generated at step (a);
d. said database training at least one neural network configured to convert at least one 2d image to a 3d model;
wherein said 3d model obtained at step (a) is an undivided 3d model, obtained directly, without a dummy model.
12. The non-transitory computer readable medium according to claim 11 additionally comprising sampling said 3d models obtained at step a.
13. The non-transitory computer readable medium according to claim 11 , wherein said rigging and skinning of said 3d model is performed by projecting said 3d model to at least one 2d image and calculating a 3d pose estimation based thereon.
14. The non-transitory computer readable medium according to claim 13 , wherein said rigging and skinning of said 3d model is performed in an automatic manner.
15. The non-transitory computer readable medium according to claim 11 , wherein at least one of said static poses is a T-pose or an A-pose.