Keyboard sensor for augmenting smart glasses sensor
A system and method for operating a keyboard and wearable extended reality appliance combination to control a virtual display are disclosed. The system may include a processor configured to receive, from a first hand-position sensor associated with the wearable extended reality appliance, first signals representing first hand-movements; receive, from a second hand-position sensor associated with the keyboard, second signals representing second hand-movements, wherein the second hand-movements include actions other than interactions with a feedback component; and control the virtual display based on the first signals and the second signals.
1. A non-transitory computer readable medium configured for use with a keyboard and wearable extended reality appliance combination in order to control a virtual display, wherein the keyboard includes an associated input area including a touch pad and keys, the computer readable medium containing instructions that when executed by at least one processor cause the at least one processor to perform operations, comprising:
receiving, from a first hand-position sensor included in the wearable extended reality appliance, first signals representing first hand-movements;
receiving, from a second hand-position sensor included in the keyboard, second signals representing second hand-movements, wherein the second hand-movements do not include physical contact with a feedback component associated with the keyboard;
detecting the second-hand movements in an area outside the input area; and
controlling the virtual display based on the first signals and the second signals.
2. The non-transitory computer readable medium of claim 1 , wherein the second hand-position sensor is an image sensor.
3. The non-transitory computer readable medium of claim 1 , wherein the second hand-position sensor is a proximity sensor.
4. The non-transitory computer readable medium of claim 1 , wherein a type of the second hand-position sensor differs from a type of the first hand-position sensor.
5. The non-transitory computer readable medium of claim 1 , wherein the operations further include:
determining an orientation of the keyboard; and
adjusting display settings associated with the virtual display based on the orientation of the keyboard.
6. The non-transitory computer readable medium of claim 1 , wherein the second hand-movements include an interaction with a no-feedback object.
7. The non-transitory computer readable medium of claim 1 , wherein when the keyboard is located on a top surface of a desk, the second hand-movements include an interaction with the top surface of a desk.
8. The non-transitory computer readable medium of claim 1 , wherein the operations further include controlling the virtual display based on the first signals and the second signals when a certainty level associated with the second hand-movements is above a threshold.
9. The non-transitory computer readable medium of claim 1 , wherein, when at least one of the first hand-movements and the second hand-movements are detected by the second hand-position sensor and undetected by the first hand-position sensor, the operations further include controlling the virtual display based only on the second signals.
10. The non-transitory computer readable medium of claim 1 , wherein, when the wearable extended reality appliance is unconnected to the keyboard, the operations further include controlling the virtual display based only on the first signals.
11. The non-transitory computer readable medium of claim 1 , wherein the wearable extended reality appliance is selectively connectable to the keyboard via a connector located on a side closest to a space bar.
12. The non-transitory computer readable medium of claim 1 , wherein the operations further include:
determining three-dimensional positions of at least part of a hand based on the first signals and the second signals; and
controlling the virtual display based on the determined three-dimensional positions of the at least part of the hand.
13. The non-transitory computer readable medium of claim 1 , wherein the operations further include:
analyzing the second signals to determine that a hand is touching a part of a physical object associated with a virtual widget;
analyzing the first signals to determine whether the hand belongs to a user of the wearable extended reality appliance;
in response to a determination that the hand belongs to the user of the wearable extended reality appliance, executing an action associated with the virtual widget; and
in response to a determination that the hand does not belong to the user of the wearable extended reality appliance, forgoing performing the action associated with the virtual widget.
14. The non-transitory computer readable medium of claim 13 , wherein the operations further include:
analyzing the second signals to determine a location where the hand is touching the physical object; and
using the determined location to select the action associated with the virtual widget.
15. The non-transitory computer readable medium of claim 1 , wherein the keyboard includes a plurality of keys, and wherein the operations further include:
analyzing the second signals to determine a user intent to press a particular key of the plurality of keys; and
causing the wearable extended reality appliance to provide a virtual indication representing the particular key based on the determined user intent.
16. The non-transitory computer readable medium of claim 1 , wherein the keyboard includes a plurality of keys, and wherein the operations further include:
analyzing the second signals to determine a user intent to press at least one of a group of keys out of the plurality of keys; and
causing the wearable extended reality appliance to provide a virtual indication representing the group of keys based on the determined user intent.
17. The non-transitory computer readable medium of claim 1 , wherein the feedback component associated with the keyboard includes a key of the keyboard or a touchpad of the keyboard.
18. A method for operating a keyboard and wearable extended reality appliance combination in order to control a virtual display, wherein the keyboard includes an associated input area including a touch pad and keys, the method comprising:
receiving, from a first hand-position sensor included in the wearable extended reality appliance, first signals representing first hand-movements;
receiving, from a second hand-position sensor included in the keyboard, second signals representing second hand-movements, wherein the second hand-movements do not include physical contact with a feedback component associated with the keyboard;
detecting the second-hand movements in an area outside the input area; and
controlling the virtual display based on the first signals and the second signals.
19. A system for operating a keyboard and wearable extended reality appliance combination in order to control a virtual display, wherein the keyboard includes an associated input area including a touch pad and keys, the system comprising:
at least one processor configured to:
receive, from a first hand-position sensor included in the wearable extended reality appliance, first signals representing first hand-movements;
receive, from a second hand-position sensor included in the keyboard, second signals representing second hand-movements, wherein the second hand-movements do not include physical contact with a feedback component associated with the keyboard;
detecting the second-hand movements in an area outside the input area; and
control the virtual display based on the first signals and the second signals.
20. A non-transitory computer readable medium configured for use with a keyboard including a plurality of feedback components and a wearable extended reality appliance in order to control a virtual display, the computer readable medium containing instructions that when executed by at least one processor cause the at least one processor to perform operations, comprising:
receiving first signals representing first hand-movements from a first hand-position sensor associated with the wearable extended reality appliance, wherein the first hand-position sensor is associated with a first field of view;
receiving second signals representing second hand-movements from a second hand-position sensor associated with the keyboard, wherein the second hand-position sensor is associated with a second field of view and the second hand-movements are outside the first field of view and do not include interactions with the plurality of feedback components of the keyboard;
receiving third signals representing two-dimensional input from at least one of the plurality of feedback components of the keyboard; and
controlling the virtual display based on the first signals, the second signals, and the third signals.