Granted Patent B2

US 12,430,778 · App. 17/596,794 · Granted Sep 30, 2025

Inventors: Abhishek Kar (Sunnyvale, CA); Hossam Isack (Oakland, CA); Adarsh Prakash Murthy Kowdle (San Francisco, CA); Aveek Purohit (Palo Alto, CA); Dmitry Medvedev (San Francisco, CA)

Assignee: GOOGLE LLC

G06T7/50G06T7/285G06T7/593G06T2207/10028G06T2207/20084

Abstract

Claims (79)

1. A method for depth estimation, the method comprising:

receiving an image frame from a sensor system of a computing device;

generating, by a model, a first depth map and first surface normals based on the image frame, the first depth map having a first scale, the first scale being a non-real-world scale;

receiving depth estimate data having values according to a second scale, the second scale being a real-world scale;

estimating at least one affine parameter based on the depth estimate data and the first depth map;

transforming the first depth map to a second depth map using the at least one affine parameter, the second depth map having the second scale;

estimating second surface normals from the second depth map;

computing a loss based on the first surface normals and the second surface normals; and

updating the model based on the loss.

2. The method of claim 1 , wherein the depth estimate data includes feature points computed from the image frame.

3. The method of claim 2 , further comprising:

detecting at least one planar region in the image frame from the feature points; and

estimating the at least one affine parameter based on the depth estimate data-estimates and the at least one planar region.

4. The method of claim 1 , wherein the at least one affine parameter is estimated based on offsets between the first depth map and the depth estimate data-estimates.

5. The method of claim 1 , further comprising:

generating a three-dimensional map from the second depth map; and

detecting at least one planar region in the image frame using the three-dimensional map, the at least one planar region configured to be used to attach a virtual object.

6. The method of claim 1 , wherein the first depth map is generated by the model using a single image frame as an input to the model.

7. The method of claim 1 , wherein the depth estimate data includes depth data from a time-of-flight sensor.

8. The method of claim 1 , wherein the depth estimate data includes depth data from a light detection and ranging sensor.

9. A method comprising:

receiving image data from a sensor system of a user device;

generating, by a model, a first depth map based on the image data, the first depth map having a first scale;

obtaining depth estimates associated with the image data;

transforming the first depth map to a second depth map using the depth estimates, the second depth map having a second scale;

predicting, by the model, first surface normals based on the image data;

predicting second surface normals from the second depth map;

computing a self-consistency loss based on the first surface normals and the second surface normals; and

updating the model based on the self-consistency loss.

10. A depth estimation system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations comprising:

receiving an image frame from a sensor system of a user device;

generating, by a model, a first depth map and first surface normals based on the image frame, the first depth map having a first scale, the first scale being a non-real-world scale;

receiving depth estimates having values according to a second scale, the second scale being a real-world scale;

estimating at least one affine parameter based on the depth estimates and the first depth map, the at least one affine parameter including at least one of shift or scale;

transforming the first depth map to a second depth map using the at least one affine parameter, the second depth map having the second scale;

estimating second surface normals from the second depth map;

computing a loss based on the first surface normals and the second surface normals; and

updating the model based on the loss.

11. The depth estimation system of claim 10 , wherein the model is configured to execute on the user device, the model including a depth-wise separable convolution, a rectified linear activation function, and a bilinear upsampling operation.

12. The depth estimation system of claim 10 , wherein the depth estimates include feature points computed from the image frame.

13. The depth estimation system of claim 10 , wherein the operations further comprise:

receiving pose data, a gravity direction, and planar region data; and

estimating the at least one affine parameter also using the pose data, the gravity direction, and the planar region data.

14. The depth estimation system of claim 10 , wherein the operations further comprise:

executing an objective function to estimate at least one of the scale or the shift.

15. The depth estimation system of claim 10 , wherein the operations further comprise:

generating a three-dimensional (3D) map based on the second depth map; and

detecting at least one planar region in the image frame by executing a plane fitting algorithm on the three-dimensional-3D map.

16. A depth estimation system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations comprising:

obtaining image data from a sensor system of a user device;

generating, by a model, a first depth map based on the image data, the first depth map having a first scale;

obtaining depth estimates associated with the image data;

estimating at least one affine parameter based on the depth estimates and the first depth map;

transforming the first depth map to a second depth map using the at least one affine parameter, the second depth map having a second scale;

predicting, using the model, first surface normals based on the image data;

predicting second surface normals from the second depth map;

computing a self-consistency loss based on the first surface normals and the second surface normals;

computing a loss based on the first surface normals and ground truth normals; and

updating the model based on the self-consistency loss and the loss.

17. A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to execute operations, the operations comprising:

receiving an image frame from a sensor system of a user device;

generating, by a model, a first depth map and first surface normals based on the image frame, the first depth map having a first scale, the first scale being a non-real-world scale;

receiving depth estimate data having values according to a second scale, the second scale being a real-world scale;

estimating at least one affine parameter based on the depth estimate data and the first depth map;

transforming the first depth map to a second depth map using the at least one affine parameter, the second depth map having the second scale, the second scale being a real-world scale;

estimating second surface normals from the second depth map;

computing a loss based on the first surface normals and the second surface normals; and

updating the model based on the loss.

18. The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise:

estimating the at least one affine parameter based on offsets between the first depth map and the depth estimate data.

19. The non-transitory computer-readable medium of claim 17 , wherein the depth estimate data includes feature points computed from the image frame.

20. The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise:

generating a three-dimensional map based on the second depth map; and

detecting at least one planar region in the image frame using the three-dimensional map, the at least one planar region configured to be used to attach a virtual object.

21. The non-transitory computer-readable medium of claim 17 , wherein the depth estimate data includes depth data from a time-of-flight sensor or a light detection and ranging sensor.

Assignments (1)

Continuity (1)

References Cited (25)