IP Library › Granted Patent US 11,727,587

Granted Patent B2

US 11,727,587 · App. 17/096,814 · Granted Aug 15, 2023

Method and system for scene image modification

Inventors: Brian Pugh (Mountain View, CA); Angus Dorbie (Redwood City, CA); Salma Jiddi (Mountain View, CA); Qiqin Dai (Mountain View, CA); Paul Gauthier (Mountain View, CA); Marc Eder (Mountain View, CA); Jianfeng Yin (Mountain View, CA); Luis Puig Morales (Mountain View, CA); Michael Otrada (Mountain View, CA); Konstantinos Nektarios Lianos (Mountain View, CA); Philip Guindi (Mountain View, CA); Brian Totty (Mountain View, CA)

Assignee: Geomagical Labs, Inc.

G06T7/543 G06T7/60 G06T7/73 G06T7/90 G06T19/20 G06T2207/10028

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 11,727,587
App. No.: 17/096,814
Filed: Nov 12, 2020
Granted: Aug 15, 2023
Kind: B2
Examiner: DESIRE, GREGORY M
Art Unit: 2664
USPC: 382/285

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

System and method for rendering virtual objects onto an image.

Claims (312)

1. A method executed by one or more computing devices of an image processing platform, the method comprising:

accessing a photorealistic image of a room;

generating at least one segmentation mask that identifies real objects included in the photorealistic image;

generating geometric surface information for the photorealistic image;

generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image, wherein generating the dense depthmap comprises:

identifying edge pixels included in edges of real objects included in the photorealistic image based at least in part on the at least one generated segmentation mask;

for at least one edge pixel, enhancing the depth for the pixel based at least in part on depths of other edge pixels included in the same edge;

identifying planar surface pixels that are included in a planar surface identified by the generated geometric surface information; and

for at least one planar surface pixel, adjusting the depth for the pixel based at least in part on depths of other planar surface pixels included in the same planar surface; and

transmitting at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device.

2. The method of claim 1 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image based at least in part on the dense depthmap, at least one segmentation mask, and the geometric surface information.

3. The method of claim 2 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

4. The method of claim 3 ,

wherein the photorealistic image is bound to a rectangular occlusion quad of the user device,

wherein the occlusion quad is associated with the dense depthmap,

wherein a fragment shader included in the user device processes image candidate pixels from the photorealistic image on the occlusion quad and virtual object candidate pixels from the virtual objects, and

wherein, for each virtual object pixel, determining the depth of the image pixel at the same location as the virtual object pixel comprises: for each virtual object candidate pixel, accessing the depth of the image candidate pixel from the occlusion quad at the same location as the virtual object pixel, by using the dense depthmap associated with the occlusion quad.

5. The method of claim 4 , further comprising:

generating an updated dense depthmap; and

wherein the end user application is configured to render at least one virtual object onto the photorealistic image based at least in part on the at least one segmentation mask, the updated dense depthmap, and the geometric surface information; and

wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

6. The method of claim 2 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

sampling the dense depthmap to generate depthmap samples;

generating a triangular occlusion mesh;

tesselating the triangular occlusion mesh, such that the depths of the pixels of the triangular mesh correspond to depths of the depthmap samples;

registering the triangular occlusion mesh with a 3D graphics system of the user device as an invisible, z-occluding scene model; and

using the 3D graphics system to render the virtual objects with occlusion by the triangular occlusion mesh by discarding occluded virtual object pixels and compositing non-occluded virtual object pixels with the photorealistic image.

7. The method of claim 2 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

identifying placeability behavior of the virtual object;

mapping two-dimensional user interface pointer coordinates to three-dimensional scene coordinates based on the placeability behavior of the virtual object and nearby scene geometry;

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, performing placement processing for the virtual object pixel.

8. The method of claim 1 , further comprising:

updating the dense depthmap, and

transmitting an updated dense depthmap to the end user application.

9. The method of claim 1 , further comprising:

receiving user selection information identifying selected portions of at least one real object to be removed from the photorealistic image; and

for each pixel in the selected portions:

estimating a new depth for the pixel and updating the dense depthmap to include the new depth, and

determining a replacement color for the pixel and updating the photorealistic image to include the determined color.

10. The method of claim 9 , further comprising:

for each real object to be removed, reducing at least one cast shadow associated with the real object to be removed,

wherein estimating a new depth for a pixel of a selected portion comprises updating the dense depthmap to include a predetermined depth as the new depth for the pixel, and

wherein reducing at least one cast shadow comprises at least one of:

inferring at least one cast shadow by performing image processing,

inferring at least one cast shadow by using a trained neural network,

inferring at least one cast shadow from detected and estimated light sources,

inferring at least one cast shadow from intrinsic image decomposition, or

inferring at least one cast shadow from plenoptic light fields.

11. The method of claim 9 , wherein estimating a new depth for a pixel of a selected portion comprises:

identifying a room architectural geometric surface that includes the location of the pixel, based at least in part on the generated geometric surface information for the photorealistic image, wherein the generated geometric surface information identifies at least one room architectural geometric surface included in the photorealistic image;

identifying at least one depth of the room architectural geometric surface, based at least in part on the generated dense depthmap;

computing a depth for the pixel based at least in part on the identified at least one depth of the room architectural geometric surface; and

updating the dense depthmap to include the computed depth as the new depth for the pixel.

12. The method of claim 9 , wherein estimating a new depth for a pixel of a selected portion comprises:

estimating a depth of an obscured geometric surface obscured by the selected portion based at least in part on the dense depthmap to identify depths of the real objects surrounding the selected portion; and

updating the dense depthmap to include the estimated depth of the obscured geometric surface as the new depth for the pixel.

13. The method of claim 9 , wherein estimating a new depth for a pixel of a selected portion comprises:

estimating a depth of an obscured geometric surface obscured by the selected portion based at least in part on trained neural networks and geometric priors from the dense depthmap; and

updating the dense depthmap to include the estimated depth of the obscured geometric surface as the new depth for the pixel.

14. The method of claim 9 , wherein determining a replacement color for a pixel of a selected portion comprises: assigning to the pixel one of: a static ghosting color, a dynamic ghosting color, a static ghosting pattern, or a dynamic ghosting pattern.

15. The method of claim 9 , wherein determining a replacement color for a pixel of a selected portion comprises:

identifying user selection of a color and assigning the selected color to the pixel,

wherein the user selection of the color relates to one of: the pixel or a patch of pixels that includes the pixel that is assigned the color.

16. The method of claim 9 , wherein determining a replacement color for a pixel of a selected portion comprises:

assigning a color of a transparency grid pattern to the pixel.

17. The method of claim 16 , wherein the transparency grid pattern is perspective warped based on inferred geometry behind the selected portion.

18. The method of claim 9 , wherein determining a replacement color for a pixel of a selected portion comprises:

performing texture synthesis to identify a texture of pixels likely surrounding the pixel;

identifying a replacement color for the pixel by using the identified texture; and

assigning the replacement color to the pixel.

19. The method of claim 9 , wherein determining a replacement color for a pixel of a selected portion comprises at least one of:

generating a replacement virtual object that includes the location of the pixel, wherein the replacement virtual object includes pixels having various colors that represent a synthetic texture; or

estimating a plenoptic light field for a scene of the photorealistic image and using the light field to re-generate an image area affected by removal of pixels included in the selected portion.

20. A method executed by one or more computing devices of an image processing platform, the method comprising:

accessing a photorealistic image of a room;

generating at least one segmentation mask that identifies real objects included in the photorealistic image;

generating geometric surface information for the photorealistic image;

generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmitting at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

wherein the end user application is configured to render at least one virtual object onto the photorealistic image based at least in part on the dense depthmap, at least one segmentation mask, and the geometric surface information by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

21. A method executed by one or more computing devices of an image processing platform, the method comprising:

accessing a photorealistic image of a room;

generating at least one segmentation mask that identifies real objects included in the photorealistic image;

generating geometric surface information for the photorealistic image;

generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmitting at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

determining a depth of the virtual object pixel;

sampling the dense depthmap to generate depthmap samples;

generating a triangular occlusion mesh;

tesselating the triangular occlusion mesh, such that the depths of the pixels of the triangular mesh correspond to depths of the depthmap samples;

registering the triangular occlusion mesh with a 3D graphics system of the user device as an invisible, z-occluding scene model; and

22. A method executed by one or more computing devices of an image processing platform, the method comprising:

accessing a photorealistic image of a room;

generating at least one segmentation mask that identifies real objects included in the photorealistic image;

generating geometric surface information for the photorealistic image;

generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmitting at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

identifying placeability behavior of the virtual object;

mapping two-dimensional user interface pointer coordinates to three-dimensional scene coordinates based on the placeability behavior of the virtual object and nearby scene geometry;

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, performing placement processing for the virtual object pixel.

23. A method executed by one or more computing devices of an image processing platform, the method comprising:

accessing a photorealistic image of a room;

generating at least one segmentation mask that identifies real objects included in the photorealistic image;

generating geometric surface information for the photorealistic image;

generating a dense depthmap that includes depth estimates for each pixel of the photorealistic image;

transmitting at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

receiving user selection information identifying selected portions of at least one real object to be removed from the photorealistic image; and

for each pixel in the selected portions:

estimating a new depth for the pixel and updating the dense depthmap to include the new depth; and

determining a replacement color for the pixel and updating the photorealistic image to include the determined color.

24. An image processing apparatus, the apparatus comprising:

one or more processors; and

one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

access a photorealistic image of a room;

generate at least one segmentation mask that identifies real objects included in the photorealistic image;

generate geometric surface information for the photorealistic image;

generate a dense depthmap that includes depth estimates for each pixel of the photorealistic image, wherein generating the dense depthmap comprises:

identifying edge pixels included in edges of real objects included in the photorealistic image based at least in part on the at least one generated segmentation mask;

for at least one edge pixel, enhancing the depth for the pixel based at least in part on depths of other edge pixels included in the same edge;

identifying planar surface pixels that are included in a planar surface identified by the generated geometric surface information; and

for at least one planar surface pixel, adjusting the depth for the pixel based at least in part on depths of other planar surface pixels included in the same planar surface; and

transmit at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device.

25. The apparatus of claim 24 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image based at least in part on the dense depthmap, at least one segmentation mask, and the geometric surface information.

26. The apparatus of claim 25 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

27. The apparatus of claim 25 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

determining a depth of the virtual object pixel;

sampling the dense depthmap to generate depthmap samples;

generating a triangular occlusion mesh;

tesselating the triangular occlusion mesh, such that the depths of the pixels of the triangular mesh correspond to depths of the depthmap samples;

registering the triangular occlusion mesh with a 3D graphics system of the user device as an invisible, z-occluding scene model; and

28. The apparatus of claim 24 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

update the dense depthmap, and

transmit an updated dense depthmap to the end user application.

29. The apparatus of claim 25 , wherein the end user application is configured to render at least one virtual object onto the photorealistic image by, for each virtual object pixel of the virtual object:

identifying placeability behavior of the virtual object;

mapping two-dimensional user interface pointer coordinates to three-dimensional scene coordinates based on the placeability behavior of the virtual object and nearby scene geometry;

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, performing placement processing for the virtual object pixel.

30. The apparatus of claim 24 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

receive user selection information identifying selected portions of at least one real object to be removed from the photorealistic image; and

for each pixel in the selected portions:

estimate a new depth for the pixel and updating the dense depthmap to include the new depth, and

determine a replacement color for the pixel and updating the photorealistic image to include the determined color.

31. An image processing apparatus, the apparatus comprising:

one or more processors; and

access a photorealistic image of a room;

generate at least one segmentation mask that identifies real objects included in the photorealistic image;

generate geometric surface information for the photorealistic image;

generate a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmit at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, displaying the image pixel.

32. An image processing apparatus, the apparatus comprising:

one or more processors; and

access a photorealistic image of a room;

generate at least one segmentation mask that identifies real objects included in the photorealistic image;

generate geometric surface information for the photorealistic image;

generate a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmit at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

determining a depth of the virtual object pixel;

sampling the dense depthmap to generate depthmap samples;

generating a triangular occlusion mesh;

tesselating the triangular occlusion mesh, such that the depths of the pixels of the triangular mesh correspond to depths of the depthmap samples;

registering the triangular occlusion mesh with a 3D graphics system of the user device as an invisible, z-occluding scene model; and

33. An image processing apparatus, the apparatus comprising:

one or more processors; and

access a photorealistic image of a room;

generate at least one segmentation mask that identifies real objects included in the photorealistic image;

generate geometric surface information for the photorealistic image;

generate a dense depthmap that includes depth estimates for each pixel of the photorealistic image; and

transmit at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

identifying placeability behavior of the virtual object;

mapping two-dimensional user interface pointer coordinates to three-dimensional scene coordinates based on the placeability behavior of the virtual object and nearby scene geometry;

determining a depth of the virtual object pixel;

determining a depth of an image pixel at the same location as the virtual object pixel;

in response to a determination that the depth of the image pixel is greater than the depth of the virtual object pixel, displaying the virtual object pixel; and

in response to a determination that the depth of the virtual object pixel is greater than the depth of the image object pixel, performing placement processing for the virtual object pixel.

34. An image processing apparatus, the apparatus comprising:

one or more processors; and

access a photorealistic image of a room;

generate at least one segmentation mask that identifies real objects included in the photorealistic image;

generate geometric surface information for the photorealistic image;

generate a dense depthmap that includes depth estimates for each pixel of the photorealistic image;

transmit at least one segmentation mask, the dense depthmap, and the geometric surface information to an end user application executed by a user device;

receive user selection information identifying selected portions of at least one real object to be removed from the photorealistic image; and

for each pixel in the selected portions:

estimate a new depth for the pixel and updating the dense depthmap to include the new depth; and

determine a replacement color for the pixel and updating the photorealistic image to include the determined color.

35. At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by one or more computing devices, cause at least one of the one or more computing devices to: