Gaze timer based augmentation of functionality of a user input device
A wearable computing system that includes a head-mounted display implements a gaze timer feature for enabling the user to temporarily extend the functionality of a handheld controller or other user input device. In one embodiment, when the user gazes at, or in the vicinity of, a handheld controller for a predetermined period of time, the functionality of one or more input elements (e.g., buttons) of the handheld controller is temporarily modified. For example, the function associated with a particular controller button may be modified to enable the user to open a particular menu using the button. The gaze timer feature may, for example, be used to augment the functionality of a handheld controller or other user input device during mixed reality and/or augmented reality sessions.
1. A computerized process performed under control of program instructions executed by one or more hardware processors, the process comprising:
detecting, based on sensor data collected during a mixed reality or augmented reality session, a gaze event in which a user gazes at, or within a defined region of, a physical object;
in response to detecting the gaze event, starting a timer used to measure whether a duration of the gaze event satisfies a condition for temporarily modifying functionality of a handheld controller associated with the mixed reality or augmented reality session; and
in response to determining that the duration of the gaze event satisfies the condition, modifying functionality of a user input element of the handheld controller, wherein modifying the functionality of the user input element comprises causing the user input element to be modified to a function that is not controllable via the handheld controller immediately before the gaze event,
wherein the modifying the function of the user input element comprises causing a button of the handhold controller to be mapped to a function of displaying a menu.
2. The computerized process of claim 1 , wherein the physical object is a hand of the user.
3. The computerized process of claim 1 , wherein, prior to modifying the functionality of the user input element, the process further comprises selecting a function modification of the user input element from a plurality of function modifications.
4. The computerized process of claim 3 , wherein the physical object is a device within a real-world environment of the mixed reality or augmented reality session.
5. The computerized process of claim 4 , wherein the selecting of the function modification of the user input element from the plurality of function modifications includes selecting the function modification is based on the device associated with the gaze event.
6. The computerized process of claim 1 , further comprising:
detecting a closing of the menu; and
in response to detecting the closing of the menu, restoring a prior function of the button of the handheld controller.
7. The computerized process of claim 1 , further comprising:
detecting a termination of the gaze event; and
in response to detecting the termination of the gaze event, restoring a prior function of the user input element of the handheld controller.
8. The computerized process of claim 1 , further comprising: displaying, on a display generated by a head mounted display worn by the user, a visual indication that the timer has been started.
9. The computerized process of claim 8 , wherein the visual indication is a textual message.
10. The computerized process of claim 1 , further comprising, in response to detecting the gaze event, generating at least one of a haptic or audible feedback signal that the timer has been started.
11. A wearable computing system, comprising:
a head-mountable display system comprising a display and one or more sensors;
a user input device configured to receive user input; and
one or more processors programmed to implement a process that comprises:
detecting, based on sensor data collected during a mixed reality or augmented reality session, a gaze event in which a user gazes at, or within a defined region of, a physical object;
in response to detecting the gaze event, starting a timer used to measure whether a duration of the gaze event satisfies a condition for temporarily modifying functionality of the user input device associated with the mixed reality or augmented reality session; and
in response to determining that the duration of the gaze event satisfies the condition, modifying functionality of a user input element of the user input device, wherein modifying the functionality of the user input element comprises causing the user input element to be modified to a function that is not controllable via the user input device immediately before the gaze event,
wherein, prior to modifying the functionality of the user input element, the process implemented by the one or more processors further comprises selecting a function modification of the user input element from a plurality of function modifications,
wherein the physical object is a device within a real-world environment of the mixed reality or augmented reality session, and
wherein the selecting of the function modification of the user input element from the plurality of function modifications includes selecting the function modification is based on the device associated with the gaze event.
12. The wearable computing system of claim 11 , wherein the modifying the function of the user input element comprises causing a button of the user input device to be mapped to a function of displaying a menu.
13. The wearable computing system of claim 12 , wherein the process implemented by the one or more processors further comprises:
detecting a closing of the menu; and
in response to detecting the closing of the menu, restoring a prior function of the button of the user input device.
14. The wearable computing system of claim 11 , wherein the process implemented by the one or more processors further comprises:
detecting a termination of the gaze event; and
in response to detecting the termination of the gaze event, restoring a prior function of the user input element of the user input device.
15. The wearable computing system of claim 11 , wherein the process implemented by the one or more processors further comprises: displaying, on a display generated by the head mounted display system that is worn by the user, a visual indication that the timer has been started.
16. The wearable computing system of claim 15 , wherein the visual indication is a textual message.
17. A computerized process performed under control of program instructions executed by one or more hardware processors, the process comprising:
detecting, based on sensor data collected during a mixed reality or augmented reality session, a gaze event in which a user gazes at, or within a defined region of, a physical object;
in response to detecting the gaze event, starting a timer used to measure whether a duration of the gaze event satisfies a condition for temporarily modifying functionality of a handheld controller associated with the mixed reality or augmented reality session; and
in response to determining that the duration of the gaze event satisfies the condition, modifying functionality of a user input element of the handheld controller, wherein modifying the functionality of the user input element comprises causing the user input element to be modified to a function that is not controllable via the handheld controller immediately before the gaze event,
wherein, prior to modifying the functionality of the user input element, the process further comprises selecting a function modification of the user input element from a plurality of function modifications,
wherein the physical object is a device within a real-world environment of the mixed reality or augmented reality session, and
wherein the selecting of the function modification of the user input element from the plurality of function modifications includes selecting the function modification is based on the device associated with the gaze event.