Embedding complex 3D objects into an augmented reality scene using image segmentation
Techniques related to embedding a 3D object model within a 3D scene are discussed. Such techniques include determining two or more object mask images for two or more corresponding cameras trained on the 3D scene, projecting 3D points from the 3D object model to the image planes of the two or more cameras, and determining a position and orientation of the 3D object model in the scene using the object mask images and the projected 3D points.
1. A system for generating a virtual view within a 3D scene comprising:
a memory to store a 3D model of a 3D object within the 3D scene; and
one or more processors coupled to the memory, the one or more processors to:
generate a first object mask image and a second object mask image, each of the first and second object mask images comprising a 2D representation of the 3D object, and the first and second object mask images representative of first and second image planes of first and second cameras trained on the 3D scene;
project, based on an initial position and orientation of the 3D model in the 3D scene, a plurality of 3D points of the 3D model onto the first and second image planes; and
determine a final position and orientation of the 3D model in the 3D scene based on optimization of a cost function that compares the first and second object mask images to the projected 3D points on the first and second image planes.
2. The system of claim 1 , wherein optimization of the cost function comprises minimization of a sum of piecewise linear functions each comprising a sum of differences between a maximum value of the first and second object mask images differenced with a value of the first and second object mask images corresponding to projections of each of the plurality of 3D points onto the first and second object mask images.
3. The system of claim 2 , wherein the projection of each of the plurality of 3D points comprises determination of a 3D location of each of the plurality of 3D points in the 3D scene using a current position and orientation of the 3D model and projection from the 3D location onto the first and second object mask images using first and second projection matrices corresponding to the first and second image planes.
4. The system of claim 1 , further comprising the one or more processors to select the initial position and orientation of the 3D model as a prior final position and orientation of the 3D model in a prior modeling of the 3D scene.
5. The system of claim 1 , wherein the one or more processors to generate the first and second object mask images comprises the one or more processors to determine first and second binary mask images comprising first pixel values for pixels deemed within the 3D object and second pixel values for pixels deemed outside of the 3D object and to dilate the first and second binary mask images to generate the first and second object mask images, wherein the first and second object mask images comprise grayscale images.
6. The system of claim 5 , wherein the one or more processors to determine the first and second binary mask images comprises application of a convolutional neural network to first and second source images attained via the first and second cameras.
7. The system of claim 5 , wherein the one or more processors to dilate comprises the one or more processors to generate an increasing gradient in the first binary mask image in a portion of the pixels outside of the object toward pixels within the object.
8. The system of claim 7 , wherein the one or more processors to dilate comprises the one or more processors to apply a first gradient or a second gradient responsive to a first initial position and orientation confidence value or a second initial position and orientation confidence value less than the first initial position and orientation confidence value, respectively, wherein the first gradient has a greater slope in pixel value change over pixel distance change than the second gradient.
9. The system of claim 1 , wherein the 3D model comprises a mesh object model comprising a plurality of indices representative of the 3D object and faces between the indices, and the plurality of 3D points comprise points from faces of the mesh model.
10. The system of claim 1 , wherein the 3D model comprises a non-rigid 3D model comprising one or more degrees of freedom for movements between 3D segments of the 3D model and the one or more processors to determine the final position and orientation of the 3D model further comprises the one or more processors to determine final parameters for the one or more degrees of freedom for the movements based on optimization of the cost function.
11. The system of claim 1 , the one or more processors to:
generate the virtual view within the 3D scene based at least in part on the final position and orientation of the 3D model in the 3D scene.
12. A method for generating a virtual view within a 3D scene comprising:
receiving a 3D model of a 3D object within the 3D scene;
generating a first object mask image and a second object mask image, each of the first and second object mask images comprising a 2D representation of the 3D object, and the first and second object mask images representative of first and second image planes of first and second cameras trained on the 3D scene;
projecting, based on an initial position and orientation of the 3D model in the 3D scene, a plurality of 3D points of the 3D model onto the first and second image planes; and
determining a final position and orientation of the 3D model in the 3D scene based on optimization of a cost function that compares the first and second object mask images to the projected 3D points on the first and second image planes.
13. The method of claim 12 , wherein optimization of the cost function comprises minimization of a sum of piecewise linear functions each comprising a sum of differences between a maximum value of the first and second object mask images differenced with a value of the first and second object mask images corresponding to projections of each of the plurality of 3D points onto the first and second object mask images.
14. The method of claim 12 , wherein said generating the first and second object mask images comprises determining first and second binary mask images comprising first pixel values for pixels deemed within the 3D object and second pixel values for pixels deemed outside of the 3D object and dilating the first and second binary mask images to generate the first and second object mask images, wherein the first and second object mask images comprise grayscale images.
15. The method of claim 14 , wherein said dilating comprises generating an increasing gradient in the first binary mask image in a portion of the pixels outside of the object toward pixels within the object and wherein said dilating comprises applying a first gradient or a second gradient responsive to a first initial position and orientation confidence value or a second initial position and orientation confidence value less than the first initial position and orientation confidence value, respectively, wherein the first gradient has a greater slope in pixel value change over pixel distance change than the second gradient.
16. The method of claim 12 , wherein the 3D model comprises a non-rigid 3D model comprising one or more degrees of freedom for movements between 3D segments of the 3D model and determining the final position and orientation of the 3D model further comprises determining final parameters for the one or more degrees of freedom for the movements based on optimization of the cost function.
17. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to generate a virtual view within a 3D scene by:
receiving a 3D model of a 3D object within the 3D scene;
generating a first object mask image and a second object mask image, each of the first and second object mask images comprising a 2D representation of the 3D object, and the first and second object mask images representative of first and second image planes of first and second cameras trained on the 3D scene;
projecting, based on an initial position and orientation of the 3D model in the 3D scene, a plurality of 3D points of the 3D model onto the first and second image planes; and
determining a final position and orientation of the 3D model in the 3D scene based on optimization of a cost function that compares the first and second object mask images to the projected 3D points on the first and second image planes.
18. The non-transitory machine readable medium of claim 17 , wherein optimization of the cost function comprises minimization of a sum of piecewise linear functions each comprising a sum of differences between a maximum value of the first and second object mask images differenced with a value of the first and second object mask images corresponding to projections of each of the plurality of 3D points onto the first and second object mask images.
19. The non-transitory machine readable medium of claim 17 , wherein said generating the first and second object mask images comprises determining first and second binary mask images comprising first pixel values for pixels deemed within the 3D object and second pixel values for pixels deemed outside of the 3D object and dilating the first and second binary mask images to generate the first and second object mask images, wherein the first and second object mask images comprise grayscale images.
20. The non-transitory machine readable medium of claim 19 , wherein said dilating comprises generating an increasing gradient in the first binary mask image in a portion of the pixels outside of the object toward pixels within the object and wherein said dilating comprises applying a first gradient or a second gradient responsive to a first initial position and orientation confidence value or a second initial position and orientation confidence value less than the first initial position and orientation confidence value, respectively, wherein the first gradient has a greater slope in pixel value change over pixel distance change than the second gradient.