Granted Patent B2

US 12,374,045 · App. 18/120,005 · Granted Jul 29, 2025

Inventors: Pavan Kumar Kamaraju (London, GB); Nikilesh Urella (Portland, OR); Flora Ponjou Tasse (London, GB)

Assignee: STREEM, LLC

G06T17/20G06T7/20G06T15/04G06T15/205G06T2200/08

Abstract

Claims (68)

1. A method for creating and texturing a 3D mesh as it is captured, comprising:

capturing, with a capturing device, a video of a three-dimensional space by moving the capturing device about the space, the video comprised of a plurality of images;

capturing, with motion sensors, motion data of the capturing device as it is moved about the space simultaneously with the captured video and obtaining depth data;

generating a 3D mesh of the space from the video and the motion data;

selecting, while the video and motion data are being captured, a plurality of keyframes from the plurality of images, each of the plurality of keyframes having an overlap with other individuals of the plurality of keyframes that is less than a predetermined threshold;

texturing, while the video and motion data are being captured, the 3D mesh with the plurality of keyframes;

detecting a warped or distorted area in the 3D mesh; and

retexturing all or a portion of the warped or distorted area with a different keyframe taken from a different perspective.

2. The method of claim 1 , wherein selecting the plurality of keyframes and texturing the 3D mesh is progressively carried out while the video of the three-dimensional space is being captured.

3. The method of claim 2 , further comprising refining the texturing as the capturing device recaptures previously textured areas.

4. The method of claim 1 , further comprising enhancing a geometry of the 3D mesh.

5. The method of claim 4 , wherein enhancing the geometry of the 3D mesh comprises:

capturing a plurality of 3D points; and

for each of the plurality of 3D points:

identifying one or more neighboring points within the 3D point's neighborhood, the neighborhood defined as a predetermined radius around the 3D point;

computing an intersection point of a normal of the 3D point and each neighboring point; and

tagging the 3D point as a curvature point when a distance between the 3D point and the intersection point is within a predetermined threshold; and

refining the 3D mesh to reflect a presence of the curvature points.

6. The method of claim 1 , wherein selecting the plurality of keyframes comprises:

determining from the motion data when the capturing device crosses a predetermined movement threshold;

capturing, when the capturing device crosses the movement threshold, an image from the video; and

determining whether the captured image is suitable for texturing.

7. The method of claim 1 , wherein selecting the plurality of keyframes comprises:

selecting for an initial keyframe a first image from the plurality of images;

adding the initial keyframe to the plurality of keyframes; and

for each image in at least a subset of the plurality of images:

determining an amount that the image overlaps each keyframe within the plurality of keyframes, and

if the amount of overlap is less than the predetermined threshold, adding the image to the plurality of keyframes.

8. The method of claim 7 , wherein determining the amount of image overlap between each image and a keyframe comprises:

creating an image plane that is contained within the keyframe, the image plane comprising a grid of 3D points;

determining how many of the 3D points are visible within the image; and

determining a percentage of visible points by dividing a number of visible points by a total number of 3D points.

9. The method of claim 1 , wherein texturing the 3D mesh comprises color correcting portions of each of the plurality of keyframes.

10. The method of claim 9 , wherein color correcting portions of each of the plurality of keyframes comprises color correcting triangles of each keyframe based on adjacent triangles.

11. A non-transitory computer-readable medium (NTCRM) containing instructions that, when executed by a processor, cause an apparatus to:

receive, from a capturing device, a video of a three-dimensional space by moving the capturing device about the space, the video comprised of a plurality of images;

receive, from motion sensors, motion data of the capturing device as it is moved about the space simultaneously with the captured video and obtaining depth data;

generating a 3D mesh of the space from the video and the motion data

select, while the video and motion data are being captured, a plurality of keyframes from the plurality of images, each of the plurality of keyframes having an overlap with other individuals of the plurality of keyframes that is less than a predetermined threshold; and

texture, while the video and motion data are being captured, the 3D mesh with the plurality of keyframes;

detecting a warped or distorted area in the 3D mesh; and

retexturing all or a portion of the warped or distorted area with a different keyframe taken from a different perspective.

12. The NTCRM of claim 11 , wherein the instructions are to cause the apparatus to select the plurality of keyframes and texture the 3D mesh progressively while the video of the three-dimensional space is being captured.

13. The NTCRM of claim 12 , wherein the instructions are to cause the apparatus to refine the texturing as additional captures of previously textured areas are received from the capturing device.

14. The NTCRM of claim 11 , wherein the instructions are to cause the apparatus to enhance a geometry of the 3D mesh.

15. The NTCRM of claim 14 , wherein to enhance the geometry of the 3D mesh, the instructions are to cause the apparatus to:

capture a plurality of 3D points; and

for each of the plurality of 3D points:

identify one or more neighboring points within the 3D point's neighborhood, the neighborhood defined as a predetermined radius around the 3D point;

compute an intersection point of a normal of the 3D point and each neighboring point; and

tag the 3D point as a curvature point when a distance between the 3D point and the intersection point is within a predetermined threshold; and

refine the 3D mesh to reflect a presence of the curvature points.

16. The NTCRM of claim 11 , wherein to select the plurality of keyframes, the instructions are to cause the apparatus to:

determine from the motion data when the capturing device crosses a predetermined movement threshold;

capture, when the capturing device crosses the movement threshold, an image from the video; and

determine whether the captured image is suitable for texturing.

17. The NTCRM of claim 11 , wherein to select the plurality of keyframes, the instructions are to cause the apparatus to:

select for an initial keyframe a first image from the plurality of images;

add the initial keyframe to the plurality of keyframes; and

for each image in at least a subset of the plurality of images:

determine an amount that the image overlaps each keyframe within the plurality of keyframes, and

if the amount of overlap is less than the predetermined threshold, add the image to the plurality of keyframes.

18. The NTCRM of claim 17 , wherein to determine the amount of image overlap between each image and a keyframe, the instructions are to cause the apparatus to:

create an image plane that is contained within the keyframe, the image plane comprising a grid of 3D points;

determine how many of the 3D points are visible within the image; and

determine a percentage of visible points by dividing a number of visible points by a total number of 3D points.

19. The NTCRM of claim 18 , wherein the grid of 3D points is a 32×32 grid of x-y-z coordinates.

20. The NTCRM of claim 11 , wherein the instructions are to further cause the apparatus to color correct portions of each of the plurality of keyframes.

Assignments (1)

Continuity (2)

References Cited (27)