Virtual manipulation of augmented and virtual reality objects
Systems and methods are provided. For example, a method includes determining a position of a user's hand and identifying a manipulation gesture performed by the user targeting a virtual object. The method also includes determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed, and determining a 3D end point based on a movement of the user's hand from the origin point. The method additionally includes deriving a 3D vector based on the 3D origin point and the 3D end point, and applying an action to the targeted virtual object based on the 3D vector, wherein the targeted virtual object is at a distance greater than the user's arm reach.
1 . A system, comprising:
one or more hardware processors; and
at least one memory storing instructions that cause the one or more hardware processors to perform operations comprising:
determining a position of a user's hand using one or more sensors;
identifying a manipulation gesture performed by the user's hand targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the 3D origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying a velocity to the targeted virtual object based on the 3D vector; and
continuing movement of the targeted virtual object at the velocity until a disengage gesture is detected.
2 . The system of claim 1 , wherein the velocity applied to the targeted virtual object has a direction corresponding to the 3D vector and a magnitude both derived via observation of the hand movement from the 3D origin point to the 3D end point.
3 . The system of claim 2 , wherein the magnitude is based on a length of the 3D vector or on a hand velocity of the user's hand from the 3D origin point to the 3D end point.
4 . The system of claim 2 , wherein applying the velocity comprises moving the targeted virtual object in an augmented reality (AR) or a virtual reality (VR) environment in the direction and at the velocity.
5 . The system of claim 4 , wherein the disengage gesture comprises a stop gesture performed by the user.
6 . The system of claim 1 , wherein the manipulation gesture comprises a hand gesture, a body gesture, a voice command, or a combination thereof.
7 . The system of claim 6 , wherein the hand gesture comprises a pinch gesture, and wherein determining the 3D end point comprises determining a point in space where the user stops moving the user's hand while maintaining the pinch gesture, and wherein the disengage gesture comprises releasing the pinch gesture.
8 . The system of claim 7 , wherein identifying the manipulation gesture comprises using an artificial intelligence model to determine that the manipulation gesture is the pinch gesture.
9 . The system of claim 8 , wherein the artificial intelligence model comprises a deep learning model configured to detect hand landmarks directly from a camera image.
10 . The system of claim 1 , comprising operations for:
determining a second 3D origin point based on a stop position of the user's hand when the user stops moving the hand;
determining a second 3D end point based on a second movement of the user's hand from the second 3D origin point;
deriving a second 3D vector based on the second 3D origin point and the second 3D end point; and
applying a second velocity to the targeted virtual object based on the second 3D vector.
11 . The system of claim 10 , wherein applying the second velocity comprises moving the targeted virtual object based on the second 3D vector, wherein the second velocity has a direction corresponding to a vector direction of the second 3D vector and a magnitude based on a length of the second 3D vector.
12 . The system of claim 1 , wherein the one or more sensors includes: camera sensors, radar sensors, light detection and ranging (lidar) sensors, or a combination thereof, included in an AR device, a VR device, or a combination thereof, worn by the user.
13 . The system of claim 12 , wherein the one or more sensors includes: camera sensors, radar sensors, lidar sensors, or a combination thereof, disposed external to the AR device and to the VR device.
14 . The system of claim 1 , wherein applying the velocity to the targeted virtual object comprises executing instructions for a joystick driver.
15 . The system of claim 14 , wherein the instructions for the joystick driver emulate a physical joystick.
16 . The system of claim 1 , comprising deriving that the user is targeting the virtual object based on identifying a targeting gesture performed by the user targeting the virtual object.
17 . The system of claim 16 , wherein the targeting gesture comprises a hand gesture, a body gesture, a voice command, or a combination thereof.
18 . The system of claim 17 , wherein the hand gesture comprises an extended palm pointing towards the targeted virtual object.
19 . A method comprising:
determining a position of a user's hand;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a position of a user's hand using one or more sensors;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying a velocity to the targeted virtual object based on the 3D vector; and
continuing movement of the targeted virtual object at the velocity, the targeted virtual object continuing at the velocity until a disengage gesture is detected.
20 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
determining a position of a user's hand using one or more sensors;
identifying a manipulation gesture performed by the user targeting a virtual object;
determining a three-dimensional (3D) origin point based on the position of the user's hand when the manipulation gesture is performed;
determining a 3D end point based on a movement of the user's hand from the origin point;
deriving a 3D vector based on the 3D origin point and the 3D end point;
applying a velocity to the targeted virtual object based on the 3D vector; and
continuing movement of the targeted virtual object at the velocity, the targeted virtual object continuing at the velocity until a disengage gesture is detected.