Granted Patent B1

US 10,950,034 · App. 16/773,850 · Granted Mar 16, 2021

Inventors: Alberto Garcia Garcia (Redmond, WA); Gioacchino Noris (Zurich, CH); Gian Diego Tipaldi (Zurich, CH)

Assignee: Facebook Technologies, LLC

G06T15/40G06T7/11G06T15/20G06T2207/10028

Abstract

Claims (88)

1. A method comprising, by a computing system:

computing, based on at least one image of a physical environment surrounding a user, depth measurements of at least one physical object;

generating a first model of the at least one physical object using the depth measurements;

identifying a plurality of first pixels in the image that depict the physical object;

associating the plurality of first pixels with a first representative depth value computed using the first model;

determining, for a pixel of an output image, that a portion of the first model and a portion of a second model of a virtual object are visible from a viewpoint;

determining that the portion of the first model is associated with the plurality of first pixels; and

rendering the output image depicting the physical object and the virtual object from the perspective of the user, wherein occlusion at the pixel is determined based on a comparison between the first representative depth value and a depth value associated with the portion of the second model.

2. The method of claim 1 , wherein the plurality of first pixels in the image are identified by an instance segmentation mask.

3. The method of claim 2 , wherein the instance segmentation mask is generated by processing the image using a machine-learning model.

4. The method of claim 1 , wherein the first representative depth value is computed by:

determining a plurality of portions of the first model that are associated with the plurality of first pixels;

determining a plurality of depth values associated with the plurality of portions of the first model; and

computing the first representative depth value based on the plurality of depth values.

5. The method of claim 1 , wherein the first representative depth value is computed by:

determining a plurality of portions of the first model that are associated with the plurality of first pixels;

determining a plurality of coordinates associated with the plurality of portions of the first model;

determining a centroid based on the plurality of coordinates; and

computing the first representative depth value based on the centroid.

6. The method of claim 1 , further comprising:

identifying a plurality of second pixels in the image that correspond to a padded boundary around the plurality of first pixels;

associating the plurality of second pixels with a second representative depth value computed based on the first representative depth value, the second representative depth value being larger than the first representative depth value;

determining, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determining that the second portion of the first model is associated with the plurality of second pixels;

wherein occlusion at the second pixel is determined based on a comparison between the second representative depth value and a second depth value associated with the second portion of the second model.

7. The method of claim 6 , wherein the plurality of second pixels are identified based on a gradient of the plurality of first pixels.

8. The method of claim 1 , further comprising:

determining, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determining that the second portion of the first model is associated with a corresponding pixel in the image that is outside of the plurality of first pixels;

wherein occlusion at the second pixel is determined based on a comparison between a second depth value associated with the second portion of the second model and a third depth value associated with the second portion of the first model.

9. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

compute, based on at least one image of a physical environment surrounding a user, depth measurements of at least one physical object;

generate a first model of the at least one physical object using the depth measurements;

identify a plurality of first pixels in the image that depict the physical object;

associate the plurality of first pixels with a first representative depth value computed using the first model;

determine, for a pixel of an output image, that a portion of the first model and a portion of a second model of a virtual object are visible from a viewpoint;

determine that the portion of the first model is associated with the plurality of first pixels; and

render the output image depicting the physical object and the virtual object from the perspective of the user, wherein occlusion at the pixel is determined based on a comparison between the first representative depth value and a depth value associated with the portion of the second model.

10. The media of claim 9 , wherein the software is further operable when executed to:

determine a plurality of portions of the first model that are associated with the plurality of first pixels;

determine a plurality of depth values associated with the plurality of portions of the first model; and

compute the first representative depth value based on the plurality of depth values.

11. The media of claim 9 , wherein the software is further operable when executed to:

determine a plurality of portions of the first model that are associated with the plurality of first pixels;

determine a plurality of coordinates associated with the plurality of portions of the first model;

determine a centroid based on the plurality of coordinates; and

compute the first representative depth value based on the centroid.

12. The media of claim 9 , wherein the software is further operable when executed to:

identify a plurality of second pixels in the image that correspond to a padded boundary around the plurality of first pixels;

associate the plurality of second pixels with a second representative depth value computed based on the first representative depth value, the second representative depth value being larger than the first representative depth value;

determine, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determine that the second portion of the first model is associated with the plurality of second pixels;

wherein occlusion at the second pixel is determined based on a comparison between the second representative depth value and a second depth value associated with the second portion of the second model.

13. The media of claim 12 , wherein the plurality of second pixels are identified based on a gradient of the plurality of first pixels.

14. The media of claim 9 , wherein the software is further operable when executed to:

determine, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determine that the second portion of the first model is associated with a corresponding pixel in the image that is outside of the plurality of first pixels;

wherein occlusion at the second pixel is determined based on a comparison between a second depth value associated with the second portion of the second model and a third depth value associated with the second portion of the first model.

15. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

compute, based on at least one image of a physical environment surrounding a user, depth measurements of at least one physical object;

generate a first model of the at least one physical object using the depth measurements;

identify a plurality of first pixels in the image that depict the physical object;

associate the plurality of first pixels with a first representative depth value computed using the first model;

determine, for a pixel of an output image, that a portion of the first model and a portion of a second model of a virtual object are visible from a viewpoint;

determine that the portion of the first model is associated with the plurality of first pixels; and

render the output image depicting the physical object and the virtual object from the perspective of the user, wherein occlusion at the pixel is determined based on a comparison between the first representative depth value and a depth value associated with the portion of the second model.

16. The system of claim 15 , wherein the one or more of the processors are further operable when executing the instructions to cause the system to:

determine a plurality of portions of the first model that are associated with the plurality of first pixels;

determine a plurality of depth values associated with the plurality of portions of the first model; and

compute the first representative depth value based on the plurality of depth values.

17. The system of claim 15 , wherein the one or more of the processors are further operable when executing the instructions to cause the system to:

determine a plurality of portions of the first model that are associated with the plurality of first pixels;

determine a plurality of coordinates associated with the plurality of portions of the first model;

determine a centroid based on the plurality of coordinates; and

compute the first representative depth value based on the centroid.

18. The system of claim 15 , wherein the one or more of the processors are further operable when executing the instructions to cause the system to:

identify a plurality of second pixels in the image that correspond to a padded boundary around the plurality of first pixels;

associate the plurality of second pixels with a second representative depth value computed based on the first representative depth value, the second representative depth value being larger than the first representative depth value;

determine, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determine that the second portion of the first model is associated with the plurality of second pixels;

wherein occlusion at the second pixel is determined based on a comparison between the second representative depth value and a second depth value associated with the second portion of the second model.

19. The system of claim 18 , wherein the plurality of second pixels are identified based on a gradient of the plurality of first pixels.

20. The system of claim 15 , wherein the one or more of the processors are further operable when executing the instructions to cause the system to:

determine, for a second pixel of the output image, that a second portion of the first model and a second portion of the second model of the virtual object are visible from the viewpoint; and

determine that the second portion of the first model is associated with a corresponding pixel in the image that is outside of the plurality of first pixels;

wherein occlusion at the second pixel is determined based on a comparison between a second depth value associated with the second portion of the second model and a third depth value associated with the second portion of the first model.

Assignments (2)

Cited By (9)

US 12,524,906 US 12,548,245 US 12,572,253 US 12,586,294 US 12,613,938 US 12,633,068 US 12,651,413 US 12,670,001 US 12,681,561