Media streaming to augmented reality glasses over a local network
The subject technology receives, by one or more hardware processors implementing a local wireless network, a request from a client device to mirror media content displayed on a screen of the client device on a wearable device. In response to the request, the subject technology causes a display of the media content in a mirroring lens of the wearable device. While the media content is being displayed in the mirroring lens of the wearable device, the subject technology tracks hand gestures of a user wearing the wearable device and viewing the media content displayed in the mirroring lens of the wearable device. The subject technology processes navigational or manipulation data based on the tracked hand gestures and sends a navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device based on the tracked hand gestures.
1. A method, comprising:
receiving, by one or more hardware processors, a request for mirrored video content, the request initiated by a user at a client device to mirror video content displayed in a screen of the client device in a mirroring lens of a wearable device, the wearable device connected to the client device by a short-range wireless connection;
in response to receiving the request, communicating with a messaging application implementing at the client device to send an instruction, via the short-range wireless connection, to a realtime service implementing at the wearable device to identify and open a connection socket, the connection socket visible only in a local wireless network;
based on a successful opening of the connection socket, by the one or more hardware processors, using the open connection to the local wireless network to direct a flow of mirrored video content from the client device to a mirroring lens at the wearable device;
while mirrored video content is being displayed in the mirroring lens of the wearable device, tracking, by the one or more hardware processors, hand gestures of a user using the wearable device to view the mirrored video content displayed in the mirroring lens of the wearable device;
processing, by the one or more hardware processors, navigational or manipulation data based on the tracked hand gestures;
sending a navigation or manipulation instruction to a lenses application to adjust a view of the mirrored video content based on the tracked hand gestures; and
causing an automatic fallback connection to a proxy local wireless network to direct a flow of mirrored video content from the client device to the lenses application in the event of an unsuccessful opening of the connection socket.
2. The method of claim 1 , wherein the mirrored video content displayed in the mirroring lens of the wearable device augments a real-world view of the user wearing the wearable device, the real-world view including a real-world view of a user's hand making the tracked hand gestures.
3. The method of claim 2 , wherein the video content displayed in the screen of the client device includes a display of an interactive user interface for receiving user inputs into the client device, and wherein the mirrored video content displayed in the mirroring lens of the wearable device includes a virtual user interface based in appearance on the interactive user interface displayed in the client device.
4. The method of claim 3 , wherein the virtual user interface displayed in the mirroring lens of the wearable device is a visual replica of the interactive user interface displayed in the screen of the client device.
5. The method of claim 4 , wherein an identified hand gesture among the hand gestures tracked by the one or more hardware processors is associated with a desired user navigation or manipulation of the virtual user interface displayed in the mirroring lens of the wearable device, and wherein a navigation or manipulation instruction based on the desired user navigation or manipulation is included in the navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device.
6. The method of claim 5 , wherein the navigation or manipulation instruction based on the identified hand gesture is associated with one or more of a hover, a resize, a reorientation, a rotation, and a repositioning of the virtual user interface displayed in the mirroring lens of the wearable device.
7. The method of claim 6 , further comprising causing or performing, by the one or more hardware processors, a navigation or manipulation of the virtual user interface displayed in the mirroring lens of the wearable device based on a received navigation or manipulation instruction.
8. The method of claim 7 , wherein the identified hand gesture includes a finger pinch hand gesture.
9. The method of claim 7 , wherein the identified hand gesture includes a single hand gesture.
10. The method of claim 7 , wherein the identified hand gesture includes a double-handed gesture.
11. A system comprising:
a processor implementing a local wireless network; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving, by one or more hardware processors, a request for mirrored video content, the request initiated by a user at a client device to mirror video content displayed in a screen of the client device in a mirroring lens of a wearable device, the wearable device connected to the client device by a short-range wireless connection;
in response to receiving the request, communicating with a messaging application implementing at the client device to send an instruction, via the short-range wireless connection, to a realtime service implementing at the wearable device to identify and open a connection socket, the connection socket visible only in a local wireless network;
based on a successful opening of the connection socket, by the one or more hardware processors, using the open connection to the local wireless network to direct a flow of mirrored video content from the client device to a mirroring lens at the wearable device;
while mirrored video content is being displayed in the mirroring lens of the wearable device, tracking, by the one or more hardware processors, hand gestures of a user using the wearable device to view the mirrored video content displayed in the mirroring lens of the wearable device;
processing, by the one or more hardware processors, navigational or manipulation data based on the tracked hand gestures;
sending a navigation or manipulation instruction to a lenses application to adjust a view of the mirrored video content based on the tracked hand gestures; and
causing an automatic fallback connection to a proxy local wireless network to direct a flow of mirrored video content from the client device to the lenses application in the event of an unsuccessful opening of the connection socket.
12. The system of claim 11 , wherein the mirrored video content displayed in the mirroring lens of the wearable device augments a real-world view of the user wearing the wearable device, the real-world view including a real-world view of a user's hand making the tracked hand gestures.
13. The system of claim 12 , wherein the video content displayed in the screen of the client device includes a display of an interactive user interface for receiving user inputs into the client device, and wherein the mirrored video content displayed in the mirroring lens of the wearable device includes a virtual user interface based in appearance on the interactive user interface displayed in the client device.
14. The system of claim 13 , wherein the virtual user interface displayed in the mirroring lens of the wearable device is a visual replica of the interactive user interface displayed in the screen of the client device.
15. The system of claim 14 , wherein an identified hand gesture among the hand gestures tracked by the one or more hardware processors is associated with a desired user navigation or manipulation of the virtual user interface displayed in the mirroring lens of the wearable device, and wherein a navigation or manipulation instruction based on the desired user navigation or manipulation is included in the navigation or manipulation instruction to the client device or a mirroring lens processor of the wearable device.
16. The system of claim 15 , wherein the navigation or manipulation instruction based on the identified hand gesture is associated with one or more of a hover, a resize, a reorientation, a rotation, and a repositioning of the virtual user interface displayed in the mirroring lens of the wearable device.
17. The system of claim 16 , wherein the operations further comprise causing or performing, by the one or more hardware processors, a navigation or manipulation of the virtual user interface displayed in the mirroring lens of the wearable device based on a received navigation or manipulation instruction.
18. The system of claim 17 , wherein the identified hand gesture includes one of a finger pinch hand gesture, a single hand gesture, or a double-handed gesture.
19. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:
receiving, by one or more hardware processors, a request for mirrored video content, the request initiated by a user at a client device to mirror video content displayed in a screen of the client device in a mirroring lens of a wearable device, the wearable device connected to the client device by a short-range wireless connection;
in response to receiving the request, communicating with a messaging application implementing at the client device to send an instruction, via the short-range wireless connection, to a realtime service implementing at the wearable device to identify and open a connection socket, the connection socket visible only in a local wireless network;
based on a successful opening of the connection socket, by the one or more hardware processors, using the open connection to the local wireless network to direct a flow of mirrored video content from the client device to a mirroring lens at the wearable device;
while mirrored video content is being displayed in the mirroring lens of the wearable device, tracking, by the one or more hardware processors, hand gestures of a user using the wearable device to view the mirrored video content displayed in the mirroring lens of the wearable device;
processing, by the one or more hardware processors, navigational or manipulation data based on the tracked hand gestures;
sending a navigation or manipulation instruction to a lenses application to adjust a view of the mirrored video content based on the tracked hand gestures; and
causing an automatic fallback connection to a proxy local wireless network to direct a flow of mirrored video content from the client device to the lenses application in the event of an unsuccessful opening of the connection socket.