Real screens in extended reality

1 . A computer-implemented method for displaying a screen in extended reality, the method comprising:

receiving, by a processor, video data of a real-world environment of a user of an extended reality device;

identifying, by the processor, a screen of a separate computing device in the video data;

generating, by the processor, a graphical outline around the screen of the separate computing device;

displaying, by the processor and on a display of the extended reality device, the screen of the separate computing device;

displaying, by the processor and on the display of the extended reality device, the graphical outline overlaid as a virtual border around the screen of the separate computing device;

determining, by the processor and when the extended reality device is set to operate in a passthrough mode, a gesture of a hand of the user;

detecting, by the processor, the gesture of the hand of the user with the graphical outline overlaid as the virtual border around the whole screen of the separate computing device; and

in response to detecting the gesture, performing, by the processor, an action associated with the gesture.

2 . The computer-implemented method of claim 1 , wherein the video data of the real-world environment of the user is received when the extended reality device is set to operate in the passthrough mode.

3 . The computer-implemented method of claim 1 , wherein identifying the screen of the separate computing device comprises:

identifying, by the processor, at least one image or pattern displayed on the screen of the separate computing device; and

based on the identification of the at least one image or pattern, uniquely identifying, by the processor, the separate computing device for pairing to the extended reality device.

4 . The computer-implemented method of claim 1 , wherein identifying the screen of the separate computing device comprises:

identifying, by the processor, a QR code displayed on the screen of the separate computing device; and

based on the identification of the QR code, uniquely identifying the separate computing device for pairing to the extended reality device.

5 . The computer-implemented method of claim 1 , wherein displaying the screen of the separate computing device and the graphical outline comprises:

determining, by the processor, a position of the screen in the display of the extended reality device;

generating, by the processor, a graphical overlay of the screen;

displaying, by the processor, the graphical overlay of the screen at the position of the screen; and

displaying, by the processor, the graphical outline around the graphical overlay of the screen.

6 . The computer-implemented method of claim 1 , wherein displaying, by the processor and on the display of the extended reality device, the graphical outline overlaid as the virtual border around the screen comprises:

displaying, by the processor and on the display of the extended reality device being set to operate in a passthrough mode, the graphical outline overlaid as the virtual border around the screen of the separate computing device, and

wherein detecting the gesture with the graphical outline comprises:

determining, by the processor, a position of the hand of the user in the video data; and

determining, by the processor, that the position of the hand in the video data overlaps a position of the graphical outline.

7 . The computer-implemented method of claim 1 , wherein the action includes moving, by the processor, the displayed screen and the graphical outline overlaid as the virtual border around the screen of the separate computing device to a new position in the display of the extended reality device being set to operate in a passthrough mode.

8 . The computer-implemented method of claim 1 , wherein the graphical outline identifies a detecting or pairing status for the separate computing device.

9 . The computer-implemented method of claim 1 , wherein the graphical outline identifies a connection status to signal that a pairing between the separate computing device and the extended reality device is successful.

10 . The computer-implemented method of claim 1 , wherein performing the action comprises repositioning the displayed screen and the graphical outline to a new position in the display of the extended reality device.

11 . A computing system for displaying screens in extended reality, comprising:

one or more processors; and

one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

receiving video data of a real-world environment of a user of an extended reality device;

identifying a screen of a separate computing device in the video data;

generating a graphical outline around the screen of the separate computing device;

displaying, on a display of the extended reality device, the screen of the separate computing device;

displaying, on the display of the extended reality device; the graphical outline overlaid as a virtual border around the screen of the separate computing device;

determining, by the one or more processors and when the extended reality device is set to operate in a passthrough mode, a gesture of a hand of the user;

detecting the gesture of the hand of the user interaction with the graphical outline overlaid as the virtual border around the whole screen of the separate computing device; and

in response to detecting the gesture, performing an action associated with the gesture.

12 . The computing system of claim 11 , wherein the video data of the real-world environment of the user is received when the extended reality device is set to operate in the passthrough mode.

13 . The computing system of claim 11 , wherein identifying the screen of the separate computing device comprises:

identifying at least one image or pattern displayed on the screen of the separate computing device; and

based on the identification of the at least one image or pattern, uniquely identifying the separate computing device for pairing to the extended reality device.

14 . The computing system of claim 11 , wherein identifying the screen of the separate computing device comprises:

identifying a QR code displayed on the screen of the separate computing device; and

based on the identification of the QR code, uniquely identifying the separate computing device for pairing to the extended reality device.

15 . The computing system of claim 11 , wherein displaying the screen of the separate computing device and the graphical outline comprises:

determining a position of the screen in the display of the extended reality device;

generating a graphical overlay of the screen;

displaying the graphical overlay of the screen at the position of the screen; and

displaying the graphical outline around the graphical overlay of the screen.

16 . The computing system of claim 11 , wherein displaying, by the processor and on the display of the extended reality device, the graphical outline overlaid as the virtual border around the screen comprises:

displaying, by the one or more processors and on the display of the extended reality device being set to operate in a passthrough mode, the graphical outline overlaid as the virtual border around the screen of the separate computing device, and

wherein detecting the gesture with the graphical outline comprises:

determining, by the one or more processors, a position of the hand of the user in the video data; and

determining, by the one or more processors, that the position of the hand in the video data overlaps a position of the graphical outline.

17 . One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:

receiving video data of a real-world environment of a user of an extended reality device;

identifying a screen of a separate computing device in the video data;

generating a graphical outline around the screen of the separate computing device;

displaying, on a display of the extended reality device, the screen of the separate computing device;

displaying, on the display of the extended reality device, the graphical outline overlaid as a virtual border around the screen of the separate computing device;

determining, when the extended reality device is set to operate in a passthrough mode, a gesture of a hand of the user;

detecting the gesture of the hand of the user with the graphical outline overlaid as the virtual border around the whole screen of the separate computing device; and

in response to detecting the gesture, performing an action associated with the gesture.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein identifying the screen of the separate computing device comprises:

identifying at least one image or pattern displayed on the screen of the separate computing device; and

based on the identification of the at least one image or pattern, uniquely identifying the separate computing device for pairing to the extended reality device.

19 . The one or more non-transitory computer-readable media of claim 17 , wherein identifying the screen of the separate computing device comprises:

identifying a QR code displayed on the screen of the separate computing device; and

based on the identification of the QR code, uniquely identifying the separate computing device for pairing to the extended reality device.

20 . The one or more non-transitory computer-readable media of claim 17 , wherein displaying the screen of the separate computing device and the graphical outline comprises:

determining a position of the screen in the display of the extended reality device;

generating a graphical overlay of the screen;

displaying the graphical overlay of the screen at the position of the screen; and

displaying the graphical outline around the graphical overlay of the screen.