IP Library Granted Patent US 11,378,798
Granted Patent B2
US 11,378,798 · App. 17/352,528 · Granted Jul 5, 2022

Surface modeling systems and methods

Inventor: Victor Ng-Thow-Hing (Los Altos, CA)
Assignee: Magic Leap, Inc.
G02B27/0093G02B27/017G02B27/0172G02B30/26G02B30/52G06F3/012G06F3/016G06F3/017G06F3/0308G06F3/0481G06F3/04812G06F3/04815G06T17/20G06T19/003G06V20/20H04N13/194H04N13/239H04N13/344H04N13/383G02B2027/014G02B2027/0178G02B2027/0187G02F1/292G06T2210/12G06V2201/12
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,378,798
App. No.
17/352,528
Granted
Jul 5, 2022
Kind
B2
Abstract

A method of generating a simulation of a physical object in an image of a physical environment. The method also includes generating a planar polygon mesh from at least the image. The method further includes extracting a boundary polygon of the planar polygon mesh. Moreover, the method includes generating a convex hull for the boundary polygon of the surface mesh. In addition, the method includes generating a minimal area oriented boundary polygon from the convex hull. The method may also include generating a maximal area oriented internal polygon (MAOBP) inside of the boundary polygon of the planar polygon mesh. The MAOBP is utilized to generate a 3-D surface model of the physical object, and the 3-D surface model is used to generate a simulation involving an interaction with the 3-D surface model of the physical object.

Claims (42)

1. A method of generating a simulation of a physical object in an image of a physical environment, the method comprising:

an imaging device obtaining an image of a physical environment having a physical object;

an image processor performing an image generating process, comprising:

generating a planar polygon mesh from the physical object from the image of the physical environment;

extracting a boundary polygon from the planar polygon mesh;

generating a convex hull for the boundary polygon; and

generating a minimal area oriented boundary polygon (MAOBP) from the convex hull, the minimal area oriented boundary polygon (MAOBP) comprising a polygon that contains the convex hull and the boundary polygon;

utilizing the MAOBP to generate a 3-D surface model of the physical object; and

utilizing the 3-D surface model to generate a simulation involving an interaction with the 3-D surface model of the physical object.

2. The method of claim 1 , wherein the simulation with the physical object is one of a simulated physical collision with the physical object, a simulated resting contact with the physical object and displaying lighting effects on the simulated physical object.

3. The method of claim 1 , further comprising, generating one or more images of the simulation involving an interaction with the 3-D surface model of the physical object.

4. The method of claim 1 , wherein the imaging device obtaining an image of a physical environment having a physical object comprises:

obtaining a 3-D point cloud corresponding to the physical environment using the imaging device, and

obtaining pose information for the imaging device; and

computing a truncated signed distance function for the 3-D point cloud using the pose information, wherein generating the planar polygon mesh comprises tessellating the truncated signed distance function.

5. The method of claim 4 , wherein the image generating process further comprises combining two smaller planar polygon meshes into one larger planar polygon mesh.

6. The method of claim 1 , wherein the image generating process further comprises obtaining a gravity vector, wherein the generated planar polygon mesh is one of substantially parallel and orthogonal to the gravity vector.

7. The method of claim 1 , wherein generating the convex hull comprises using a Graham-Scan algorithm.

8. The method of claim 1 , wherein generating the minimal area oriented boundary polygon (MAOBP) comprises using a rotating calipers algorithm.

9. The method of claim 1 , wherein the image generating process further comprises generating a maximal area oriented internal polygon (MAOIP) that is entirely contained inside of the boundary polygon.

10. The method of claim 9 , wherein generating the maximal area oriented internal polygon (MAOIP) comprises performing a search in a search area defined by the boundary polygon, and

wherein generating the maximal area oriented internal polygon (MAOIP) comprises forming a grid in the search area, and

the image generating process further comprises adjusting a resolution of the grid based on a size of the search area.

11. The method of claim 9 , wherein the image generating process further comprises:

receiving a selection of a point inside of the boundary polygon of the planar polygon mesh wherein generating the maximal area oriented internal polygon (MAOIP) comprises performing a search in a search area defined using the selected point; and

adjusting a resolution of the grid based on a size of the search area, and

wherein generating the maximal area oriented internal polygon (MAOIP) comprises forming a grid in the search area.

12. The method of claim 9 , wherein the minimal area oriented boundary polygon (MAOBP) and the maximal area oriented internal polygon (MAOIP) have a same shape.

13. The method of claim 1 , wherein the planar polygon mesh is generated based on a marching cubes algorithm.

14. The method of claim 1 , wherein the minimal area oriented boundary polygon (MAOBP) is at least one of a rectangle, a triangle, and a circle.

15. The method of claim 1 , further comprising determining a fit between the minimal area oriented boundary polygon (MAOBP) and the boundary polygon, and

wherein determining the fit comprises calculating a difference between a first area of the minimal area oriented boundary polygon (MAOBP) and a second area of the boundary polygon.

16. The method of claim 1 , wherein the image generating process further comprises storing data representing the minimal area oriented boundary polygon (MAOBP),

wherein the minimal area oriented boundary polygon (MAOBP) is a rectangle,

wherein the data comprises four sets of coordinates corresponding to the rectangle, and

wherein each of the four sets of coordinates is a pair of coordinates.

17. The method of claim 16 , wherein the data further comprises a length of the rectangle, a width of the rectangle, and a center of the rectangle.

18. The method of claim 1 , wherein generating the planar polygon mesh comprises capturing static portions of a series of images of the physical environment.

19. The method of claim 1 , wherein the minimal area oriented boundary polygon (MAOBP) comprises a smallest polygon that contains the convex hull and the boundary polygon, and

wherein the minimal area oriented boundary polygon (MAOBP) comprises a smallest rectangle that contains the convex hull and the boundary polygon.

20. The method of claim 19 , further comprising generating a maximal area oriented internal polygon (MAOIP) that is contained inside of the boundary polygon.

21. The method of claim 20 , wherein the minimal area oriented boundary polygon (MAOBP) overestimates an extent of a surface being modeled, and the maximal area oriented internal polygon (MAOIP) underestimates the extent of the surface being modeled.

Assignments (3)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
SECURITY INTEREST Recorded Sep 23, 2022
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 061197/0942 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2021
From: NG-THOW-HING, VICTOR
To: MAGIC LEAP, INC.
Reel/Frame 056632/0343 →