Mitigation techniques for degradation states at an artificial reality system
Aspects of the present disclosure are directed to detecting and mitigating a degradation state at an artificial reality system. An artificial reality (XR) system can comprise several different input channels. The performance of one or more of these input channels, or channel fidelity, can degrade over time, such as in the presence of different operating conditions. Implementations can perform mitigation workflows to improve the channel fidelities of various input channels. When channel fidelity is improved, the input channel may be maintained as an active channel for receiving one or more types of user input. When channel fidelity is not improved, one or more other input channel(s) can be selected as the active channel(s) (e.g., replacement input channels) for receiving one or more types of user input.
1 . A method comprising:
detecting, for a first input channel of a plurality of input channels, a channel fidelity that fails to meet a fidelity criteria, the plurality of input channels comprising two or more of an eye tracking input channel, wrist electromyography (EMG) signal input channel, computer vision hand tracking input channel, hand-held controller input channel, and trackpad touch input channel, wherein the plurality of input channels provide user input from a user to an artificial reality system; and
in response:
performing a mitigation workflow configured to improve the channel fidelity for the first input channel, the mitigation workflow comprising a guided user workflow, a calibration workflow for the first input channel, or a combination thereof; and
selecting one or more primary input channels from among the plurality of input channels,
wherein the first input channel is selected as the one or more primary input channels when it is determined that the channel fidelity of the first input channel meets the fidelity criteria after performance of the mitigation workflow, and
wherein one or more of the plurality of input channels other than the first input channel are selected as the one or more primary input channels when it is determined that the channel fidelity of the first input channel fails to meet the fidelity criteria after performance of the mitigation workflow.
2 . The method of claim 1 , wherein an ordered priority is predefined for the plurality of input channels, and the one or more of the plurality of input channels other than the first input channel are selected as the one or more primary input channels according to the ordered priority.
3 . The method of claim 1 , wherein the first input channel comprises the eye tracking input channel and the mitigation workflow comprises an eye tracking calibration workflow.
4 . The method of claim 3 , wherein the artificial reality system displays an artificial reality environment to the user, and the eye tracking calibration workflow comprises:
receiving an eye tracking input relative to one or more known locations of one or more user interface elements displayed to the user via the artificial reality system;
resolving the eye tracking input;
selecting one of the one or more user interface elements with the one or more known locations in response to the resolved eye tracking input being a first distance from a known location of the selected user interface element; and
correcting one or more calibration parameters using the first distance as an estimated error.
5 . The method of claim 4 , wherein the one or more calibration parameters are corrected in response to the user continuing to interact with the artificial reality system without reverting the selection of the one of the one or more user interface elements.
6 . The method of claim 4 , wherein the eye tracking calibration workflow comprises a passive calibration workflow that operates in a background without explicit user instruction to select the one of the one or more user interface elements.
7 . The method of claim 1 , wherein the mitigation workflow comprises a guided user workflow that instructs the user to adjust eyeglasses worn by the user.
8 . The method of claim 1 , wherein the one or more of the plurality of input channels other than the first input channel selected as the one or more primary input channels comprises at least the computer vision hand tracking input channel.
9 . The method of claim 1 , wherein the one or more of the plurality of input channels other than the first input channel selected as the one or more primary input channels comprises at least the trackpad touch input channel, the hand-held controller input channel, or any combination thereof.
10 . The method of claim 1 , wherein the selected one or more primary input channels control interactions between the user and an artificial reality environment displayed to the user via the artificial reality system.
11 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process, the process comprising:
detecting, for a first input channel of a plurality of input channels, a channel fidelity that fails to meet a fidelity criteria, the plurality of input channels comprising two or more of an eye tracking input channel, wrist electromyography (EMG) signal input channel, computer vision channel hand tracking input channel, hand-held controller input channel, trackpad touch input channel, or any combination thereof, wherein the plurality of input channels provide user input from a user to an artificial reality system; and
in response:
performing a mitigation workflow configured to improve the detected channel fidelity, the mitigation workflow comprising a guided user workflow, a calibration workflow for the first input channel, or a combination thereof; and
selecting one or more primary input channels from among the plurality of input channels,
wherein the first input channel is selected when it is determined that the channel fidelity meets the fidelity criteria after performance of the mitigation workflow, and
wherein one or more of the plurality of input channels other than the first input channel are selected when it is determined that the channel fidelity fails to meet the fidelity criteria after performance of the mitigation workflow.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein an ordered priority is predefined for the plurality of input channels, and one or more input channels other than the first input channel are selected as the one or more primary input channels according to the ordered priority.
13 . The non-transitory computer-readable storage medium of claim 11 , wherein the first input channel comprises an eye tracking input channel and the mitigation workflow comprises an eye tracking calibration workflow.
14 . The non-transitory computer-readable storage medium of claim 13 , wherein the artificial reality system displays an artificial reality environment to the user, and performance of the eye tracking calibration workflow comprises:
receiving an eye tracking input relative to one or more known locations of one or more user interface elements displayed to the user via the artificial reality system;
resolving the eye tracking input;
selecting one of the one or more user interface elements with the one or more known locations in response to the resolved eye tracking input being a first distance from a known location of the selected user interface element; and
correcting one or more calibration parameters using the first distance as an estimated error.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein the one or more calibration parameters are corrected in response to the user continuing to interact with the artificial reality system without reverting the selection of the one of the one or more user interface elements.
16 . The non-transitory computer-readable storage medium of claim 14 , wherein the eye tracking calibration workflow comprises a passive calibration workflow that operates in a background without explicit user instruction to select the one of the one or more user interface elements.
17 . The non-transitory computer-readable storage medium of claim 13 , wherein the mitigation workflow comprises a guided user workflow that instructs the user to adjust eyeglasses worn by the user.
18 . The non-transitory computer-readable storage medium of claim 11 , wherein the one or more input channels selected as the one or more primary input channels comprise at least a computer vision hand tracking input channel.
19 . The non-transitory computer-readable storage medium of claim 11 , wherein the one or more input channels selected as the one or more primary input channels comprise a trackpad touch input, a hand-held controller input channel, or any combination thereof.
20 . A computing system, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
detecting, for a first input channel of a plurality of input channels, a channel fidelity that fails to meet a fidelity criteria, the plurality of input channels comprising two or more of an eye tracking input channel, wrist electromyography (EMG) signal input channel, computer vision channel hand tracking input channel, hand-held controller input channel, trackpad touch input channel, or any combination thereof, wherein the plurality of input channels provide input to an artificial reality system from a user; and
in response:
performing a mitigation workflow configured to improve the channel fidelity for the first input channel, the mitigation workflow comprising a guided user workflow, a calibration workflow for the first input channel, or a combination thereof; and
selecting one or more primary input channel from among the plurality of input channels, wherein,
the first input channel is selected when it is determined that the channel fidelity meets the fidelity criteria after performance of the mitigation workflow, and
one or more of the plurality of input channels other than the first input channel are selected when it is determined that the channel fidelity fails to meet the fidelity criteria after performance of the mitigation workflow.