Overlaying 3D augmented reality content on real-world objects using image segmentation

1. A computer-implemented method, comprising:

determining, by a processor, a position and a scale of a physical object depicted in an image from a viewpoint;

identifying, by the processor, a symmetry-based mismatch between a three-dimensional (3D) model of the physical object and a digital outline of the physical object at the viewpoint, the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object;

causing, by the processor, the 3D model of the physical object to match the position and the scale of the physical object based on the symmetry-based mismatch;

overlaying, by the processor, a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and

generating, by the processor, a composite view of the 3D virtual object overlaid on the exterior portion of the physical object, wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint, wherein the 3D virtual object is an exterior object and a portion of the 3D virtual object is omitted from the composite view based on the portion of the 3D virtual object being partially obstructed by the physical object at the viewpoint.

2. The computer-implemented method of claim 1 , wherein the predetermined locational relation defines where the 3D virtual object belongs on the 3D model, the method further comprising:

displaying the composite view of the 3D virtual object overlaid on the exterior portion of the physical object on a display.

3. The computer-implemented method of claim 1 , further comprising:

applying, by the processor, a machine learning (ML) model to the digital outline, wherein the machine learning model is trained on a two-dimensional (2D) outline of a three-dimensional (3D) model of the physical object at a plurality of rotations;

extracting, by the ML model, the 2D outline of the 3D model at each of the plurality of rotations of the physical object;

comparing, by the ML model, the 2D outline of the 3D model to the digital outline of the physical object at each of the plurality of rotations of the physical object; and

determining, by the ML model, a match between the 2D outline of the 3D model and the digital outline at a first rotation of the plurality of rotations of the physical object.

4. The computer-implemented method of claim 3 , wherein the position and the scale of the physical object is determined based on the match between the 2D outline of the 3D model and the digital outline at the first rotation of the plurality of rotations of the physical object.

5. The computer-implemented method of claim 3 , wherein the 3D virtual object is rotated according to a first rotation of a plurality of rotations of the 3D virtual object, wherein the 3D virtual object at the first rotation of the plurality of rotations of the 3D virtual object matches the position and scale of the physical object at the first rotation of the plurality of rotations of the physical object.

6. The computer-implemented method of claim 3 , wherein the ML model comprises a guess-and-check model, wherein a count of the plurality of rotations of the physical object is based on a predetermined count of guesses and checks applied by the guess-and-check model.

7. The computer-implemented method of claim 1 , further comprising:

detecting, by the processor, a plane in the image;

detecting, by the processor, a floor in the image based on the plane detection; and

wherein the position of the physical object is based at least in part on one or more feature points on the floor and a location of the 3D model of the physical object relative to a virtual floor corresponding to the floor.

8. The computer-implemented method of claim 1 , further comprising:

partitioning, by the processor, the image into a plurality of segments;

applying, by the processor, a respective label to each of a plurality of pixels in each of the plurality of segments;

locating, by the processor, the physical object based on the labels; and

determining, by the processor, the digital outline of the physical object based on the labels.

9. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a processor, cause the processor to:

determine a position and a scale of a physical object depicted in an image from a viewpoint;

identify a symmetry-based mismatch between a three-dimensional (3D) model of the physical object and a digital outline of the physical object at the viewpoint, the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object;

cause the 3D model of the physical object to match the position and the scale of the physical object based on the symmetry-based mismatch;

overlay a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and

generate a composite view of the 3D virtual object overlaid on the exterior portion of the physical object, wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint, wherein the 3D virtual object is an exterior object and a portion of the 3D virtual object is omitted from the composite view based on the portion of the 3D virtual object being partially obstructed by the physical object at the viewpoint.

10. The computer-readable storage medium of claim 9 , wherein the predetermined locational relation defines where the 3D virtual object belongs on the 3D model, wherein the instructions further cause the processor to:

display the composite view of the 3D virtual object overlaid on the exterior portion of the physical object on a display.

11. The computer-readable storage medium of claim 9 , wherein the instructions further cause the processor to:

apply a machine learning (ML) model to the digital outline, wherein the machine learning model is trained on a two-dimensional (2D) outline of a three-dimensional (3D) model of the physical object at a plurality of rotations;

extract, by the ML model, the 2D outline of the 3D model at each of the plurality of rotations of the physical object;

compare, by the ML model, the 2D outline of the 3D model to the digital outline of the physical object at each of the plurality of rotations of the physical object; and

determine, by the ML model, a match between the 2D outline of the 3D model and the digital outline at a first rotation of the plurality of rotations of the physical object.

12. The computer-readable storage medium of claim 11 , wherein the position and the scale of the physical object is determined based on the match between the 2D outline of the 3D model and the digital outline at the first rotation of the plurality of rotations of the physical object, wherein the 3D virtual object is rotated according to a first rotation of a plurality of rotations of the 3D virtual object, wherein the 3D virtual object at the first rotation of the plurality of rotations of the 3D virtual object matches the position and scale of the physical object at the first rotation of the plurality of rotations of the physical object.

13. The computer-readable storage medium of claim 9 , wherein the instructions further cause the processor to:

detect, by the processor, a plane in the image;

detect, by the processor, a floor in the image based on the plane detection; and

wherein the position of the physical object is based at least in part on one or more feature points on the floor and a location of the 3D model of the physical object relative to a virtual floor corresponding to the floor.

14. The computer-readable storage medium of claim 9 , wherein the instructions further cause the processor to:

partition, by the processor, the image into a plurality of segments;

apply, by the processor, a respective label to each of a plurality of pixels in each of the plurality of segments;

locate, by the processor, the physical object based on the labels; and

determine, by the processor, the digital outline of the physical object based on the labels.

15. A computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, cause the processor to:

determine a position and a scale of a physical object depicted in an image from a viewpoint;

identify a symmetry-based mismatch between a three-dimensional (3D) model of the physical object and a digital outline of the physical object at the viewpoint, the symmetry-based mismatch based at least in part on symmetrical characteristics of the physical object;

cause the 3D model of the physical object to match the position and the scale of the physical object based on the symmetry-based mismatch;

overlay a 3D virtual object at a first location on an exterior portion of the physical object in augmented reality based on a predetermined locational relation between the 3D virtual object and the 3D model of the physical object; and

generate a composite view of the 3D virtual object overlaid on the exterior portion of the physical object, wherein the 3D virtual object is partially obstructed from view by the physical object at the viewpoint, wherein the 3D virtual object is an exterior object and a portion of the 3D virtual object is omitted from the composite view based on the portion of the 3D virtual object being partially obstructed by the physical object at the viewpoint.

16. The computing apparatus of claim 15 , wherein the predetermined locational relation defines where the 3D virtual object belongs on the 3D model, wherein the instructions further cause the processor to:

display the composite view of the 3D virtual object overlaid on the exterior portion of the physical object on a display.

17. The computing apparatus of claim 15 , wherein the instructions further cause the processor to:

apply a machine learning (ML) model to the digital outline, wherein the machine learning model is trained on a two-dimensional (2D) outline of a three-dimensional (3D) model of the physical object at a plurality of rotations;

extract, by the ML model, the 2D outline of the 3D model at each of the plurality of rotations of the physical object;

compare, by the ML model, the 2D outline of the 3D model to the digital outline of the physical object at each of the plurality of rotations of the physical object; and

determine, by the ML model, a match between the 2D outline of the 3D model and the digital outline at a first rotation of the plurality of rotations of the physical object.

18. The computing apparatus of claim 17 , wherein the position and the scale of the physical object is determined based on the match between the 2D outline of the 3D model and the digital outline at the first rotation of the plurality of rotations of the physical object, wherein the 3D virtual object is rotated according to a first rotation of a plurality of rotations of the 3D virtual object, wherein the 3D virtual object at the first rotation of the plurality of rotations of the 3D virtual object matches the position and scale of the physical object at the first rotation of the plurality of rotations of the physical object.

19. The computing apparatus of claim 15 , wherein the instructions further cause the processor to:

detect, by the processor, a plane in the image;

detect, by the processor, a floor in the image based on the plane detection; and

wherein the position of the physical object is based at least in part on one or more feature points on the floor and a location of the 3D model of the physical object relative to a virtual floor corresponding to the floor.

20. The computing apparatus of claim 15 , wherein the instructions further cause the processor to:

partition, by the processor, the image into a plurality of segments;

apply, by the processor, a respective label to each of a plurality of pixels in each of the plurality of segments;

locate, by the processor, the physical object based on the labels; and

determine, by the processor, the digital outline of the physical object based on the labels.