Granted Patent B2

US 11,024,088 · App. 16/304,672 · Granted Jun 1, 2021

Inventors: Simon Heinen (Aachen, DE); Lars Tholen (Aachen, DE); Mostafa Akbari-Hochberg (San Francisco, CA); Gloria Indra Dhewani Abidin (Aachen, DE)

Assignee: HoloBuilder, Inc.

G06T19/006G06T15/04G06T17/20

Abstract

Claims (58)

1. A method of creating an augmented reality scene, comprising,

by a computing device with a processor and memory,

receiving image data over a network, the image data being generated from a camera including multiple frames;

estimating a depth map of a keyframe of the multiple frames of the received image data using estimated depth values of pixels in the keyframe;

generating a point cloud using the estimated depth map of the keyframe; and

generating a 3D mesh using the generated point cloud;

wherein the 3D mesh includes multiple keyframe images, wherein the multiple keyframe images are overlapping keyframes merged to a single texture;

wherein the keyframe is a frame with a depth map and a position;

for non-keyframe frames, calculating a relative position to a keyframe using the depth map and position of the keyframe; and

refining the keyframe depth map;

wherein the 3D mesh is generated by,

computing a normal vector for each point in the point cloud, based on neighboring points;

orienting the computed normal vector of each point toward the camera pose of the keyframe that the point belongs to.

2. The method of claim 1 wherein the merging of multiple keyframes to a single texture is by weighting a keyframe distance to a mesh surface and a keyframe angle relative to the mesh surface.

3. The method of claim 2 , further comprising, by the computer,

receiving a second image data over the network,

receiving tracking markers for the second image data;

receiving objects for the second image data;

causing display of, the image, the second image data and the received objects using the tracking markers.

4. A system for creating an augmented reality scene, comprising,

a computing device with a processor and memory, configured to,

receive image data over a network, the image data being generated from a camera including multiple frames;

estimate a depth map of a keyframe of the multiple frames of the received image data using estimated depth values of pixels in the keyframe;

generate a point cloud using the estimated depth map of the keyframe; and

generate a 3D mesh using the generated point cloud;

wherein the 3D mesh includes multiple keyframe images, wherein the multiple keyframe images are overlapping keyframes merged to a single texture;

wherein the keyframe is a frame with a depth map and a position;

wherein the computer is further configured to,

for non-keyframe frames, calculate a relative position to a keyframe using the depth map and position of the keyframe; and

refine the keyframe depth map;

wherein the 3D mesh is generated by, the computer further configured to,

compute a normal vector for each point in the point cloud, based on neighboring points;

determine an oriented computed normal vector of each point toward the camera pose of the keyframe that the point belongs to.

5. The system of claim 4 wherein the merging of multiple keyframes to a single texture is by weighting a keyframe distance to a mesh surface and a keyframe angle relative to the mesh surface.

6. The method of claim 5 , wherein the computer further configured to,

receive a second image data over the network,

receive tracking markers for the second image data;

receive objects for the second image data;

cause display of, the image, the second image data and the received objects using the tracking markers.

7. A non-transitory computer-readable medium having computer-executable instructions thereon for a method of creating an augmented reality scene, the method comprising:

by a computing device with a processor and memory,

receiving image data over a network, the image data being generated from a camera including multiple frames;

estimating a depth map of a keyframe of the multiple frames of the received image data using estimated depth values of pixels in the keyframe;

generating a point cloud using the estimated depth map of the keyframe; and

generating a 3D mesh using the generated point cloud;

wherein the 3D mesh includes multiple keyframe images, wherein the multiple keyframe images are overlapping keyframes merged to a single texture;

wherein the keyframe is a frame with a depth map and a position;

for non-keyframe frames, calculating a relative position to a keyframe using the depth map and position of the keyframe; and

refining the keyframe depth map;

wherein the 3D mesh is generated by,

computing a normal vector for each point in the point cloud, based on neighboring points;

orienting the computed normal vector of each point toward the camera pose of the keyframe that the point belongs to.

8. The non-transitory computer-readable medium of claim 7 , wherein the merging of multiple keyframes to a single texture is by weighting a keyframe distance to a mesh surface and a keyframe angle relative to the mesh surface.

9. The non-transitory computer-readable medium of claim 8 , further comprising, by the computer,

receiving a second image data over the network,

receiving tracking markers for the second image data;

receiving objects for the second image data;

causing display of, the image, the second image data and the received objects using the tracking markers.

Assignments (2)

Continuity (2)

Cited By (9)

US 12,208,171 US 12,243,381 US 12,254,738 US 12,311,275 US 12,450,975 US 12,548,399 US 12,580,783 US 12,597,212 US 12,651,509