Granted Patent B2

US 12,260,476 · App. 17/590,724 · Granted Mar 25, 2025

Inventors: Christopher M. Harrises (Nashua, NH); Nicole Elizabeth Samec (Fort Lauderdale, FL); Nastasja U. Robaina (Coconut Grove, FL); Mark Baerenrodt (Millbrae, CA); Adam Carl Wright (Fort Lauderdale, FL); Adrian Kaehler (Los Angeles, CA)

Assignee: MAGIC LEAP, INC.

G06T11/60G02B27/0172G06T19/006G02B2027/014G02B2027/0178

Abstract

Claims (77)

1. A method for sharing an image in an augmented reality telepresence session, the method comprising:

establishing a bidirectional communication between a first augmented reality device associated with a first caller and a second augmented reality device associated with a second caller, the first augmented reality device comprising a first outward facing camera configured to image a region in front of the first augmented reality device;

detecting presence of a reflective surface in a region in front of the first augmented reality device, wherein the detecting the presence of the reflective surface includes keypoints recognition of characteristics of the region in front of the first augmented reality device and one or more depth discontinuities within the reflective surface, wherein detecting the presence of the reflective surface comprises:

identifying a first set of keypoints based on the keypoints recognition of characteristics of the region in front of the first augmented reality device, the first set of keypoints including a first triplet of neighboring keypoints;

maintaining a map of an environment around the first augmented reality device, the map comprising a second set of keypoints corresponding to objects in the environment, the second set of keypoints including a second triplet of neighboring keypoints;

determining geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and

matching the geometric relationships associated with the first triplet of neighboring keypoints and the second triplet of neighboring keypoints;

confirming the presence of the reflective surface based on the matching of the geometric relationships;

capturing, with the first outward facing camera, a first image of the first caller reflected from the reflective surface;

identifying a region of the first image of the first caller that includes an image of the first augmented reality device occluding a portion of a face of the first caller;

generating an unoccluded image of the first caller by substituting the identified region with a corresponding region of an image of the unoccluded face of the first caller without the image of the first augmented reality device that previously occluded the portion of the face of the first caller;

transmitting at least a portion of the unoccluded image of the first caller to the second augmented reality device; and

the determining the geometric relationships between the first triplet of neighboring keypoints and the second triplet of neighboring keypoints includes at least one of a) determining open angles among the first triplet of neighboring keypoints and open angles among the second triplet of neighboring keypoints, b) axially transforming a first geometry represented by the first triplet of neighboring keypoints or axially transforming a second geometry represented by the first triplet of neighboring keypoints, or c) determining an amount of magnification between a first size of the first geometry and a second size of the second geometry.

2. The method of claim 1 , further comprising:

receiving, by the first augmented reality device, a second image of the second caller; and

displaying at least a portion of the second image of the second caller by the first augmented reality device.

3. The method of claim 2 , wherein displaying at least a portion of the second image of the second caller comprises displaying the second image so that it occludes the first image of the first caller in the reflective surface.

4. The method of claim 1 , wherein transmitting the portion of the unoccluded image of the first caller to the second augment reality device comprises:

transmitting the unoccluded image of the first caller to a server computer system;

processing the unoccluded image of the first caller by the server computer system; and

transmitting the unoccluded image of the first caller to the second augmented reality device.

5. The method of claim 1 , further comprising processing the first image of the first caller by performing at least one of the following:

applying virtual content to the first image of the first caller; cropping a portion of the first image of the first caller; applying a background to the first image of the first caller; resizing the first image of the first caller; and

identifying a region of a body of the first caller that is not captured by the first augmented reality device and adding the identified region to the first image of the first caller.

6. The method of claim 5 , wherein the virtual content comprises blending effects, feathering effects, glow effects, or special features effects.

7. The method of claim 5 , wherein the background contains elements associated with games, movies, books, art, music, existing physical elements, or virtual objects.

8. The method of claim 1 , wherein the first augmented reality device comprises an eye-tracking camera configured to image an eye of the first caller, and wherein substituting the identified region with the corresponding region of the image of the unoccluded face of the first caller comprises inserting an image of the eye of the first caller in the identified region.

9. The method of claim 1 , wherein the unoccluded image of the first caller comprises an image patch obtained before the augmented reality telepresence session.

10. The method of claim 1 , wherein generating the unoccluded image of the first caller comprises dynamically deforming an image patch of the unoccluded face of the first caller based on data obtained from an inward-facing imaging system of the first augmented reality device.

11. The method of claim 1 , wherein the establishing the bidirectional communication connection automatically initiated in response to detecting the presence of the reflective surface.

12. An augmented reality system for conducting a teleconference between a first caller and a second caller, the system comprising:

a first augmented reality device associated with a first caller, the first augmented reality device comprising a first outward facing camera configured to image a region in front of the first augmented reality device; and

a computer processor configured to communicate with a network and programmed to:

establish a bidirectional communication between the first augmented reality device and a second augmented reality device associated with a second caller;

detect presence of a reflective surface in a region in front of the first augmented reality device, including: maintaining a map of an environment around the first augmented reality device, the map comprising a first set of keypoints corresponding to objects in the environment; identifying a second set of keypoints in the images captured by the first outward facing camera; determining geometric relationships between the first set of keypoints and between the second set of keypoints; matching the geometric relationships associated with the first set of keypoints and the second set of keypoints; and confirming the presence of the reflective surface based on the matching of the geometric relationships, wherein the first set of keypoints includes a first triplet of neighboring keypoints and the second set of keypoints includes a second triplet of neighboring keypoints,

wherein the computer processor is programmed to detect the presence of the reflective surface, including: determine the geometric relationships between the first set of keypoints and between the second set of keypoints by determining the geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and match the geometric relationships associated with the first set of keypoints and the second set of keypoints by matching the geometric relationship associated with the first triplet of neighboring keypoints and the second triplet of neighboring keypoints, and

wherein determining the geometric relationship comprises at least one of a) determining open angles among the first triplet of keypoints and open angles among the second triplet of keypoints, b) axially transforming a first geometry represented by the first set of keypoints or axially transforming a second geometry represented by the second set of keypoints, or c) determining an amount of magnification between a first size of the first geometry and a second size of the second geometry;

capture, with the first outward facing camera, a first image of the first caller reflected from the reflective surface;

identify a region of the first image of the first caller that includes an image of the first augmented reality device occluding a portion of a face of the first caller;

generate an unoccluded image of the first caller by substituting the identified region with a corresponding region of an image of an unoccluded face of the first caller without the first augmented reality device; and

transmit at least a portion of the unoccluded image of the first caller to the second augmented reality device;

receive, by the first augmented reality device, a second image of the second caller; and

display at least a portion of the second image of the second caller by the first augmented reality device.

13. The system of claim 12 , wherein the computer processor is further programmed to display at least a portion of the second image of the second caller by the first augmented reality device so that the second image of the second caller occludes the first image of the first caller in the reflective surface.

14. The system of claim 12 , wherein the computer processor is further programed to process the first image of the first caller by the first augmented reality device to perform at least one of:

apply virtual content to the first image of the first caller;

crop a portion of the first image of the first caller;

apply a background to the first image of the first caller;

resize the first image of the first caller; and

identify a region of a body of the first caller that is not captured by the first augmented reality device and add the identified region to the first image of the first caller.

15. The system of claim 12 , wherein the first augmented reality device comprises an eye-tracking camera configured to image an eye of the first caller, and wherein substituting the identified region with the corresponding region of the image of the unoccluded face of the first caller comprises inserting an image of the eye of the first caller in the identified region.

16. The system of claim 12 , wherein establishing the bidirectional communication is automatically initiated in response to detecting the presence of the reflective surface.

17. An augmented reality system for conducting a teleconference, the system comprising:

a computer system configured to communicate with a first augmented reality device associated with a first caller and a second augmented reality device associated with a second caller, the computer system comprising:

a network interface;

a non-transitory data storage; and

a hardware processor configured to communicate with the network interface and the non-transitory data storage, the processor programmed to:

detect presence of a reflective surface in front of the first augmented reality device, including: maintaining a map of an environment around the first augmented reality device, the map comprising a first set of keypoints corresponding to objects in the environment; identifying a second set of keypoints in the images captured by the a first outward facing camera of the first augmented reality device; determining geometric relationships between the first set of keypoints and between the second set of keypoints; and matching the geometric relationships associated with the first set of keypoints and the second set of keypoints; and confirming the presence of the reflective surface based on the matching of the geometric relationships

wherein the first set of keypoints includes a first triplet of neighboring keypoints and the second set of keypoints includes a second triplet of neighboring keypoints,

wherein the processor is further programmed to detect the presence of the reflective surface, including: determine the geometric relationships between the first set of keypoints and between the second set of keypoints by determining the geometric relationships between the first triplet of neighboring keypoints and between the second triplet of neighboring keypoints; and match the geometric relationships associated with the first set of keypoints

and the second set of keypoints by matching the geometric relationship associated with the first triplet of neighboring keypoints and the second triplet of neighboring keypoints, and

wherein determining the geometric relationship comprises at least one of a) determining open angles among the first triplet of keypoints and open angles among the second triplet of keypoints, b) axially transforming a first geometry represented by the first set of keypoints or axially transforming a second geometry represented by the second set of keypoints, or c) determining an amount of magnification between a first size of the first geometry and a second size of the second geometry;

capture, by the first augmented reality device from the reflective surface, a first image of the first caller reflected from the reflective surface; identify a region of the first image of the first caller that includes an image of the first augmented reality device occluding a portion of a face of the first caller;

generate an unoccluded image of the first caller by substituting the identified region with a corresponding region of an image of an unoccluded face of the first caller without the first augmented reality device;

receive a first message from the first augmented reality device, the first message comprising a request to establish a bidirectional communication with the second augmented reality device;

establish the bidirectional communication between the first and the second augmented reality device;

receive a second message from the first augmented reality device, the second message comprising the unoccluded image of the first caller;

transmit the second message to the second augmented reality device; receive a third message from the second augmented reality device,

the third message comprising an image of the second caller captured by the second augmented reality device from a reflective surface in front of the second augmented reality device; and

transmit the third message to the first augmented reality device.

18. The system of claim 17 , wherein at least one of the first augmented reality device or the second augmented reality device comprises a light field display configured to present images at multiple depth planes.

19. The system of claim 17 ,

wherein the image of the second caller comprises a plurality of patches and the image of the second caller comprises a portion of a face of the second caller unoccluded by the second augmented reality device, and

wherein the processor is programmed to further: receive a patch of an eye region of the second caller which is occluded by the second augmented reality device, and wherein to generate the image of the second caller, the processor is programmed to synthesize the image of the second caller based on the portion of the second caller's face unoccluded by the second augmented reality device and the patch of the eye region occluded by the second augmented reality device.

20. The system of claim 17 , wherein the processor is programmed to further:

receive an indication of a movement of an eye region of the second caller; access a face model of the second caller; and

update the image of the second caller based on the movement of the eye region and the face model.

Assignments (5)

Continuity (11)

References Cited (252)