IP Library Granted Patent US 10,657,716
Granted Patent B2
US 10,657,716 · App. 15/914,692 · Granted May 19, 2020

Collaborative augmented reality system

Inventors: Matthew C. Clausen (Los Angeles, CA); Charles Goddard (Los Angeles, CA); Garrett K. Johnson (Pasadena, CA); Marsette A. Vona, III (Los Angeles, CA); Victor X. Luo (Pasadena, CA); Jeffrey S. Norris (Saratoga, CA); Anthony J. Valderrama (Pasadena, CA)
Assignee: CALIFORNIA INSTITUTE OF TECHNOLOGY
G06T19/003G06F3/017G06T19/006G06T19/20G06F3/0482G06T2210/21G06T2219/004G06T2219/024G06T2219/2016
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,657,716
App. No.
15/914,692
Filed
Mar 7, 2018
Granted
May 19, 2020
Kind
B2
Art Unit
2623
USPC
345/158
Abstract

A method, apparatus, and system provide the ability to control navigation of a three-dimensional (3D) computer aided design (CAD) model in an augmented reality space. The 3D CAD model is rendered in the augmented reality space and appears as if it is present in a physical space at true scale. A virtual camera is defined as fixed to a current pose of a user's head. A virtual line segment S is constructed coincident with a ray R from a center of projection P of the virtual camera and a center pixel of the virtual camera. A check for geometric intersections between the virtual line segment S and surfaces of scene elements is conducted. Upon intersecting with a part of the model, a gaze cursor is rendered at an intersection point C closest to the center of projection P.

Claims (89)

1. A method for controlling navigation of a three-dimensional (3D) computer aided design (CAD) model in an augmented reality space comprising:

(a) rendering the 3D CAD model in the augmented reality space, wherein the 3D CAD model appears as if the 3D CAD model is present in a physical space at true scale; and

(b) for each rendered frame of the rendering:

(1) defining a virtual camera fixed to a current pose of a user's head;

(2) constructing a virtual line segment S coincident with a ray R from a center of projection P of the virtual camera and a center pixel of the virtual camera wherein the virtual line segment S starts at a pre-defined minimum distance from the projection P and ends at a pre-defined maximum distance from the projection P;

(3) checking for geometric intersections between the virtual line segment S and surfaces of scene elements, wherein the scene elements comprise one or more parts of the 3D CAD model and user interface objects;

(4) determining that there is a geometric intersection with one or more parts of the 3D CAD model; and

(5) rendering a gaze cursor at an intersection point C closest to the center of projection P, wherein the gaze cursor comprises a visually distinguishable indicator.

2. The method of claim 1 , wherein the virtual camera faces forward and represents a field of view that is rendered to a user's display.

3. The method of claim 1 , wherein the visually distinguishable indicator comprises a colored sphere.

4. The method of claim 1 , wherein the visually distinguishable indicator comprises a point light that creates a surface glow effect.

5. The method of claim 1 , further comprising:

receiving an air tap gesture; and

sending a point of interest placed network message to all participants of a session, wherein during the session, all participants are enabled to collaboratively visualize and manipulate the 3D CAD model in the augmented reality space.

6. The method of claim 5 , further comprising:

receiving the point of interest placed network message, wherein the point of interest placed network message comprises a part K, an intersection point C that is closest to the center of projection P, and a surface normal N;

highlighting the part K; and

rendering a point of interest marker at intersection point C oriented along the surface normal N.

7. The method of claim 6 , further comprising:

receiving a point of interest hidden network message;

removing the highlighting of part K; and

hiding the point of interest marker.

8. The method of claim 1 , further comprising:

for each rendered frame of the rendering:

checking for additional geometric intersections between the virtual line segment S and surfaces of scene elements;

determining that no geometric intersections are found; and

rendering the gaze cursor on ray R at a predefined distance D from the center of projection P.

9. The method of claim 1 , further comprising:

determining the intersection point C closest to the center of projection P;

classifying a type of object that intersection point C lies on as the part of the CAD model.

10. The method of claim 1 , further comprising:

determining the intersection point C closest to the center of projection P;

classifying a type of object that intersection point C lies on as a user interface object O; and

sending a gazed-at message to the user interface object O.

11. A system for controlling navigation of a three-dimensional (3D) computer aided design (CAD) model in an augmented reality space:

(a) a holographic computing platform comprising a processor and memory;

(b) a 3D CAD model rendered in the augmented reality space by the holographic computing platform, wherein the 3D CAD model appears as if the 3D CAD model is present in a physical space at true scale, wherein for each rendered frame of the rendering, the holographic computing platform:

(1) defines a virtual camera fixed to a current pose of a user's head;

(2) constructs a virtual line segment S coincident with a ray R from a center of projection P of the virtual camera and a center pixel of the virtual camera;

(3) checks for geometric intersections between the virtual line segment S and surfaces of scene elements, wherein the scene elements comprise one or more parts of the 3D CAD model and user interface objects, wherein the virtual line segment S starts at a pre-defined minimum distance from the projection P and ends at a pre-defined maximum distance from the projection P;

(4) determines that there is a geometric intersection with one or more parts of the 3D CAD model; and

(5) renders a gaze cursor at an intersection point C closest to the center of projection P, wherein the gaze cursor comprises a visually distinguishable indicator.

12. The system of claim 11 , wherein the virtual camera faces forward and represents a field of view that is rendered to a user's display.

13. The system of claim 11 , wherein the visually distinguishable indicator comprises a colored sphere.

14. The system of claim 11 , wherein the visually distinguishable indicator comprises a point light that creates a surface glow effect.

15. The system of claim 11 , wherein the holographic computing platform further:

receives an air tap gesture; and

sends a point of interest placed network message to all participants of a session, wherein during the session, all participants are enabled to collaboratively visualize and manipulate the 3D CAD model in the augmented reality space.

16. The system of claim 15 , wherein the holographic computing platform further:

receives the point of interest placed network message, wherein the point of interest placed network message comprises a part K, an intersection point C that is closest to the center of projection P, and a surface normal N;

highlights the part K; and

renders a point of interest marker at intersection point C oriented along the surface normal N.

17. The system of claim 16 , wherein the holographic computing platform further:

receives a point of interest hidden network message;

removes the highlighting of part K; and

hides the point of interest marker.

18. The system of claim 11 , wherein the holographic computing platform further:

for each rendered frame of the rendering:

checks for additional geometric intersections between the virtual line segment S and surfaces of scene elements;

determines that no geometric intersections are found; and

renders the gaze cursor on ray R at a predefined distance D from the center of projection P.

19. The system of claim 11 , wherein the holographic computing platform further:

determines the intersection point C closest to the center of projection P;

classifies a type of object that intersection point C lies on as the part of the CAD model.

20. The system of claim 11 , wherein the holographic computing platform further:

determines the intersection point C closest to the center of projection P;

classifies a type of object that intersection point C lies on as a user interface object O; and

sends a gazed-at message to the user interface object O.

21. A method for controlling navigation of a three-dimensional (3D) computer aided design (CAD) model in an augmented reality space comprising:

(a) rendering the 3D CAD model in the augmented reality space, wherein the 3D CAD model appears as if the 3D CAD model is present in a physical space at true scale;

(b) for each rendered frame of the rendering:

(1) defining a virtual camera fixed to a current pose of a user's head;

(2) constructing a virtual line segment S coincident with a ray R from a center of projection P of the virtual camera and a center pixel of the virtual camera;

(3) checking for geometric intersections between the virtual line segment S and surfaces of scene elements, wherein the scene elements comprise one or more parts of the 3D CAD model and user interface objects;

(4) determining that there is a geometric intersection with one or more parts of the 3D CAD model; and

(5) rendering a gaze cursor at an intersection point C closest to the center of projection P, wherein the gaze cursor comprises a visually distinguishable indicator;

(c) receiving an air tap gesture; and

(d) sending a point of interest placed network message to all participants of a session, wherein during the session, all participants are enabled to collaboratively visualize and manipulate the 3D CAD model in the augmented reality space.

22. A system for controlling navigation of a three-dimensional (3D) computer aided design (CAD) model in an augmented reality space:

(a) a holographic computing platform comprising a processor and memory;

(b) a 3D CAD model rendered in the augmented reality space by the holographic computing platform, wherein the 3D CAD model appears as if the 3D CAD model is present in a physical space at true scale, wherein for each rendered frame of the rendering, the holographic computing platform:

(1) defines a virtual camera fixed to a current pose of a user's head;

(2) constructs a virtual line segment S coincident with a ray R from a center of projection P of the virtual camera and a center pixel of the virtual camera;

(3) checks for geometric intersections between the virtual line segment S and surfaces of scene elements, wherein the scene elements comprise one or more parts of the 3D CAD model and user interface objects;

(4) determines that there is a geometric intersection with one or more parts of the 3D CAD model; and

(5) renders a gaze cursor at an intersection point C closest to the center of projection P, wherein the gaze cursor comprises a visually distinguishable indicator; and

(c) the holographic computing platform further:

(1) receives an air tap gesture; and

(2) sends a point of interest placed network message to all participants of a session, wherein during the session, all participants are enabled to collaboratively visualize and manipulate the 3D CAD model in the augmented reality space.

Assignments (5)
LICENSE Recorded Aug 6, 2018
From: CALIFORNIA INSTITUTE OF TECHNOLOGY
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 046729/0673 →
LICENSE Recorded Aug 6, 2018
From: CALIFORNIA INSTITUTE OF TECHNOLOGY
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 046729/0741 →
LICENSE Recorded Aug 6, 2018
From: CALIFORNIA INSTITTUTE OF TECHNOLOGY
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 046729/0830 →
LICENSE Recorded Aug 6, 2018
From: CALIFORNIA INSTITUTE OF TECHNOLOGY
To: UNITED STATES OF AMERICA AS REPRESENTED BY THE ADMINISTRATOR OF NASA
Reel/Frame 046730/0095 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 7, 2018
From: CLAUSEN, MATTHEW C.; GODDARD, CHARLES; JOHNSON, GARRETT K.; VONA, III, MARSETTE A.; LUO, VICTOR X.; NORRIS, JEFFREY S.; VALDERRAMA, ANTHONY J.
To: CALIFORNIA INSTITUTE OF TECHNOLOGY
Reel/Frame 045138/0751 →
Continuity (1)
Related Publication 20190279424A1 · Sep 12, 2019
Cited By (4)
US 12,200,409 US 12,293,450 US 12,387,434 US 12,554,221