IP Library Granted Patent US 11,402,964
Granted Patent B1
US 11,402,964 · App. 17/170,839 · Granted Aug 2, 2022

Integrating artificial reality and other computing devices

Inventors: Jasper Stevens (London, GB); Alexander Michael Louie (Milpitas, CA); Michal Hlavac (Seattle, WA); Benjamin Taylor Koga-Winn (Vancouver, WA); Wai Leong Chak (London, GB)
Assignee: Facebook Technologies, LLC
G06F3/04815G06F3/017G06F3/0486G06F16/9558G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,402,964
App. No.
17/170,839
Granted
Aug 2, 2022
Kind
B1
Abstract

Methods and systems for integrating functionality of XR devices and non-XR devices are disclosed herein. An XR device can identify and obtain one or more content items being displayed on non-XR devices and create virtual objects associated with the content item in an artificial reality environment of the XR device. A user of the XR device can interact with the virtual objects, such as by moving the virtual objects to new locations within the artificial reality environment or interact with the virtual objects to view the content items. The user of the XR device can also provide one or more gestures or commands to send content items or identifiers of content items associated with the virtual objects to non-XR devices for display at the non-XR devices.

Claims (58)

1. A method for displaying a virtual object on an XR device, the method comprising:

creating registration between A) the XR device, and B) a first non-XR device and a second non-XR device, wherein creating registration between the XR device and the first non-XR device comprises creating a first communication channel between the XR device and the first non-XR device, wherein creating registration between the XR device and the second non-XR device comprises creating a second communication channel between the XR device and the second non-XR device, and wherein no direct communication channel is created between the first and second non-XR devices;

identifying, based on a first user input to the XR device, a selection of the first non-XR device;

using the first communication channel created during registration to transfer, to the XR device, one or more content items, associated with the first non-XR device and indicated by the first user input;

creating a virtual object associated with each of the one or more identified content items;

displaying, by the XR device, the virtual object associated with each of the one or more identified content items in an artificial reality environment associated with the XR device; and

sending, by the XR device to the second non-XR device, the one or more identified content items associated with the virtual object, without direct communication between the first and second non-XR devices,

wherein the sending causes display, by the second non-XR device, of the one or more identified content items associated with the virtual object.

2. The method of claim 1 , wherein identifying the selection of the first non-XR device includes detecting a gesture of a user of the XR device directed toward the first non-XR device.

3. The method of claim 1 , wherein identifying the selection of the first non-XR device includes selecting an audio output hardware device or selecting a virtual object displayed in the artificial reality environment associated with a detected audio output of the first non-XR device.

4. The method of claim 1 , the method further comprising:

receiving the one or more identified content items at the XR device, sent from the first non-XR device over the first communication channel.

5. The method of claim 1 , wherein the one or more content items are identified by:

detecting a second user gesture directed to the first non-XR device within the artificial reality environment; and

identifying a point on a screen of the first non-XR device, the point associated with the user gesture;

wherein an indication of the point is transmitted, by the XR device to the first non-XR device over the first communication channel,

wherein the first non-XR device identifies the one or more content items being output by the first non-XR device and associated with the point; and

wherein the identifying the one or more content items associated with the first non-XR device comprises receiving, from the first non-XR device over the first communication channel, the one or more identified content items associated with the point or identifiers for the one or more content items associated with the point.

6. The method of claim 5 , wherein the first non-XR device identifies the one or more content items by:

identifying two or more embedded content items associated with the point, with at least one of the two or more embedded content items embedded within another of the two or more embedded content items;

accessing one or more rules defining a hierarchy of content item types and/or content item levels; and

evaluating the one or more accessed rules to:

select one of the one or more of content embedded content items as the one or more content items.

7. A computing system for transferring a content item to an XR device as a virtual object, the computing system comprising:

one or more processors, and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

creating registration between A) the XR device and B) a first non-XR device and a second non-XR device, wherein creating registration between the XR device and the first non-XR device comprises creating a first communication channel between the XR device and the first non-XR device, wherein creating registration between the XR device and the second non-XR device comprises creating a second communication channel between the XR device and the second non-XR device, and wherein no direct communication channel is created between the first and second non-XR devices, wherein the registration causes the one or more content item identifiers to be displayed in the artificial reality environment when the XR device is in communicative contact with the first and second non-XR devices over the first and second communication channels;

identifying a location of a user selection within an artificial reality environment of the XR device;

identifying one or more content item identifiers A) displayed relative to a corresponding content item on the first non-XR device and B) that is in relation to the location of the user selection, wherein each of the one or more content item identifiers includes a QR code associated with the corresponding content item;

obtaining the corresponding content item based on the one or more content item identifiers, wherein obtaining the corresponding content item includes retrieving the corresponding content item from the first non-XR device based on the associated QR code;

creating a virtual object associated with the obtained content item; and

displaying, by the XR device, the virtual object in the artificial reality environment of the XR device.

8. The computing system of claim 7 , wherein the QR code specifies a source of the content item.

9. The computing system of claim 7 , wherein the QR code includes a URL associated with the content item.

10. The computing system of claim 7 , wherein identifying the content item further comprises capturing an image of the content item and applying image recognition or image matching technologies to the captured image.

11. A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a process, the process comprising:

creating registration between A) the XR device and B) a first non-XR device and a second non-XR device, wherein creating registration between the XR device and the first non-XR device comprises creating a first communication channel between the XR device and the first non-XR device, wherein creating registration between the XR device and the second non-XR device comprises creating a second communication channel between the XR device and the second non-XR device, and wherein no direct communication channel is created between the first and second non-XR devices;

identifying a selection of a virtual object, displayed in an artificial reality environment of the XR device, associated with a content item retrieved from the first non-XR device over the first communication channel;

identifying a selection of the second non-XR device; and

using the second communication channel to send the content item associated with the selected virtual object to the second non-XR device;

wherein the sending of the content item to the second non-XR device causes display by the second non-XR device of the content item on a display of the second non-XR device.

12. The computer-readable storage medium of claim 11 , wherein identifying the selection of the virtual object and identifying the selection of the second non-XR device includes identifying a gesture of a user of the XR device in relation to the virtual object and the second non-XR device.

13. The computer-readable storage medium of claim 12 , wherein the gesture is a drag-and-drop gesture of a graphical representation of the virtual object within the artificial reality environment of the XR device.

14. The computer-readable storage medium of claim 13 , wherein the drag-and-drop gesture includes moving the virtual object to be positioned relative to the second non-XR device.

15. The computer-readable storage medium of claim 11 , wherein the process further comprises:

transmitting an identifier of the virtual object to the second non-XR device from the XR device over the second communication channel.

16. The computer-readable storage medium of claim 11 , wherein the identifier of the virtual object includes a QR code, and wherein causing the second non-XR device to display a content item comprises:

causing the second non-XR device to access the content item using the QR code.

17. The computer-readable storage medium of claim 11 , wherein sending the content item to the second non-XR device includes:

identifying a gesture of a user directed at the second non-XR device;

identifying an endpoint of the gesture within the artificial reality environment;

access one or more rules associated the endpoint of the gesture; and

based on the one or more accessed rules,

causing the second non-XR device to display the content item associated with the selected virtual object according to the one or more accessed rules.

18. The computer-readable storage medium of claim 17 , wherein the one or more rules are accessed by:

identifying a location within the artificial reality environment of the endpoint of the gesture;

identifying a point on a screen of the second non-XR device associated with the location of the endpoint; and

accessing the one or more rules based on the point on the screen associated with the location of the endpoint.

Assignments (2)
CHANGE OF NAME Recorded Jun 15, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060386/0364 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 2, 2021
From: STEVENS, JASPER; LOUIE, ALEXANDER MICHAEL; HLAVAC, MICHAL; KOGA-WINN, BENJAMIN TAYLOR; CHAK, WAI LEONG
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 057050/0483 →
Cited By (5)
US 12,423,924 US 12,586,326 US 12,614,355 US 12,712,840 US 12,719,882