Multi-mode two-hand gesture tracking
Various implementations disclosed herein include devices, systems, and methods that interpret user activity involving both hands based on determining an interaction mode associated with each hand of a user. For example, a process may receive data corresponding to user activity involving two hands of a user in a 3-dimensional (3D) coordinate system. The process may further identify actions performed by the two hands based on the data corresponding to the user activity. Each of the two hands performs one of the identified actions. The process may further determine whether the identified actions satisfy a criterion based on the data corresponding to the user activity and in accordance, determine to associate each of the actions performed by each of the two hands with a direct interaction mode or an indirect interaction mode.
1 . A method comprising:
at a device having a processor and one or more sensors:
receiving data corresponding to user activity involving two hands of a user in a 3D coordinate system;
identifying actions performed by the two hands based on the data corresponding to the user activity, each of the two hands performing one of the identified actions; and
in accordance with a determination that the identified actions satisfy a criterion based on the data corresponding to the user activity,
associating a first action of the actions performed by a first hand of the two hands with a first interaction mode based on a proximity of the first hand with respect to a first virtual object within the 3D coordinate system, wherein the first hand of the user directly interacts with the first virtual object in the first interaction mode; and
associating a second action of the actions performed by a second hand of the of the two hands with a second interaction mode based on a hand gesture performed by the second hand in combination with a user gaze location determined based on a convergence point of a first ray extending from a first eye of the user and a second ray extending from a second eye of the user, wherein the second hand of the user in combination with the user gaze location indirectly interacts with a second virtual object in the second interaction mode, wherein the first virtual object differs from the second virtual object, and wherein the first interaction mode occurs during a same time period as the second interaction mode.
2 . The method of claim 1 further comprising initiating a response to the user activity based on the first interaction mode and the second interaction mode.
3 . The method of claim 1 , wherein the first virtual object is a collider corresponding to a 3D virtual object.
4 . The method of claim 1 , wherein the first virtual object is a 2 dimensional (2D) UI element of an application positioned within the 3D coordinate system by a shared simulation system.
5 . The method of claim 1 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on user motion.
6 . The method of claim 1 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on user context.
7 . The method of claim 1 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on UI element positioning with respect to user gaze.
8 . The method of claim 1 , further comprising:
providing feedback indicating a direct interaction mode status and an indirect interaction mode status associated with each of the two hands.
9 . A system comprising:
a non-transitory computer-readable storage medium; and
one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
receiving data corresponding to user activity involving two hands of a user in a 3D coordinate system;
identifying actions performed by the two hands based on the data corresponding to the user activity, each of the two hands performing one of the identified actions; and
in accordance with a determination that the identified actions satisfy a criterion based on the data corresponding to the user activity,
associating a first action of the actions performed by a first hand of the two hands with a first interaction mode based on a proximity of the first hand with respect to a first virtual object within the 3D coordinate system, wherein the first hand of the user directly interacts with the first virtual object in the first interaction mode; and
associating a second action of the actions performed by a second hand of the of the two hands with a second interaction mode based on a hand gesture performed by the second hand in combination with a user gaze location determined based on a convergence point of a first ray extending from a first eye of the user and a second ray extending from a second eye of the user, wherein the second hand of the user in combination with the user gaze location indirectly interacts with a second virtual object in the second interaction mode, wherein the first virtual object differs from the second virtual object, and wherein the first interaction mode occurs during a same time period as the second interaction mode.
10 . The system of claim 9 , wherein the operations further comprise initiating a response to the user activity based on the first interaction mode and the second interaction mode.
11 . The system of claim 9 , wherein the first virtual object is a collider corresponding to a 3D virtual object.
12 . The system of claim 9 , wherein the first virtual object is a 2 dimensional (2D) UI element of an application positioned within the 3D coordinate system by a shared simulation system.
13 . The system of claim 9 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on user motion.
14 . The system of claim 9 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on user context.
15 . The system of claim 9 , wherein the first interaction mode or the second interaction mode is determined in response to predicting user intent based on UI element positioning with respect to user gaze.
16 . The system of claim 9 , further comprising:
providing feedback indicating a direct interaction mode status and an indirect interaction mode status associated with each of the two hands.
17 . A non-transitory computer-readable storage medium storing program instructions executable via one or more processors to perform operations comprising:
receiving data corresponding to user activity involving two hands of a user in a 3D coordinate system;
identifying actions performed by the two hands based on the data corresponding to the user activity, each of the two hands performing one of the identified actions; and
in accordance with a determination that the identified actions satisfy a criterion based on the data corresponding to the user activity,
associating a first action of the actions performed by a first hand of the two hands with a first interaction mode based on a proximity of the first hand with respect to a first virtual object within the 3D coordinate system, wherein the first hand of the user directly interacts with the first virtual object in the first interaction mode; and
associating a second action of the actions performed by a second hand of the of the two hands with a second interaction mode based on a hand gesture performed by the second hand in combination with a user gaze location determined based on a convergence point of a first ray extending from a first eye of the user and a second ray extending from a second eye of the user, wherein the second hand of the user in combination with the user gaze location indirectly interacts with a second virtual object in the second interaction mode, wherein the first virtual object differs from the second virtual object, and wherein the first interaction mode occurs during a same time period as the second interaction mode.
18 . The non-transitory computer-readable storage medium of claim 17 , wherein the operations further comprise initiating a response to the user activity based on the first interaction mode and the second interaction mode.