Method and system for hand tracking in a robotic system

1. A robotic system comprising:

a hand tracking system; and

a controller coupled to the hand tracking system;

wherein the controller is configured to:

receive, from the hand tracking system, a plurality of locations of a hand;

determine if the hand is in a first hand pose based on the plurality of locations;

in response to determining that the hand is in the first hand pose, switch the robotic system to detecting hand trajectories while the robotic system is operating in a first mode of operation of a plurality of modes of operation; and

while detecting the hand trajectories:

detect, based on hand tracking information from the hand tracking system, that the hand has performed a first hand trajectory of a plurality of known hand trajectories; and

in response to detecting the first hand trajectory, change a mode of operation of the robotic system to a second mode of operation of the plurality of modes of operation, wherein the second mode of operation is selected from the group consisting of: a mode of operation for controlling an associated instrument to follow motions of an input device, a mode of operation decoupling control of the associated instrument from the input device, a mode of operation for controlling an imaging device, a mode of operation for swapping the input device from control of the associated instrument to control of a second instrument, and a mode of operation for toggling display of an auxiliary video window on a display.

2. The robotic system of claim 1 , wherein the hand tracking information comprises a sequence of velocities of the hand.

3. The robotic system of claim 2 , wherein to detect the first hand trajectory, the controller is configured to compare the sequence of velocities to sequences of velocities associated with each of the plurality of known hand trajectories.

4. The robotic system of claim 2 , wherein to detect the first hand trajectory, the controller is configured to:

convert the sequence of velocities to a sequence of symbols; and

compare the sequence of symbols to sequences of symbols associated with each of the plurality of known hand trajectories.

5. The robotic system of claim 2 , wherein the sequence of velocities is a sequence of unit velocities.

6. The robotic system of claim 1 , wherein to determine that the hand is in the first hand pose, the controller is configured to match the plurality of locations to the first hand pose using a classifier.

7. The robotic system of claim 1 , wherein to determine that the hand is in the first hand pose, the controller is configured to:

generate a feature set from the plurality of locations; and

compare the feature set to feature sets of a plurality of hand poses.

8. The robotic system of claim 1 , wherein the change in mode of operation comprises enabling a display mode, disabling a display mode, or switching to a new display mode.

9. The robotic system of claim 1 , wherein the change in mode of operation comprises a change in an illumination level of a worksite.

10. The robotic system of claim 1 , wherein the change in mode of operation comprises a change to: a clutching mode, an imaging device control mode, an imaging device focus mode, or a manipulator arm swapping mode.

11. A method comprising:

receiving, by a controller of a robotic system and from a hand tracking system, a plurality of locations of a hand;

determining, by the controller, if the hand is in a first hand pose based on the plurality of locations;

in response to determining that the hand is in the first hand pose, switching, by the controller, to detecting hand trajectories while the robotic system is operating in a first mode of operation of a plurality of modes of operation; and

while detecting the hand trajectories:

detecting, based on hand tracking information from the hand tracking system, that the hand has performed a first hand trajectory of a plurality of known hand trajectories; and

in response to detecting the first hand trajectory, changing a mode of operation of the robotic system to a second mode of operation of the plurality of modes of operation, wherein the second mode of operation is selected from the group consisting of: a mode of operation for controlling an associated instrument to follow motions of an input device, a mode of operation decoupling control of the associated instrument from the input device, a mode of operation for controlling an imaging device, a mode of operation for swapping the input device from control of the associated instrument to control of a second instrument, and a mode of operation for toggling display of an auxiliary video window on a display.

12. The method of claim 11 , wherein the hand tracking information comprises a sequence of unit velocities of the hand.

13. The method of claim 12 , wherein detecting the first hand trajectory comprises comparing the sequence of unit velocities to sequences of unit velocities associated with each of the plurality of known hand trajectories.

14. The method of claim 12 , wherein detecting the first hand trajectory comprises:

converting the sequence of unit velocities to a sequence of symbols; and

comparing the sequence of symbols to sequences of symbols associated with each of the plurality of known hand trajectories.

15. The method of claim 11 , wherein determining that the hand is in the first hand pose comprises matching the plurality of locations to the first hand pose using a classifier.

16. The method of claim 11 , wherein changing the mode of operation comprises:

enabling a display mode;

disabling a display mode; or

switching to a new display mode.

17. A non-transitory tangible medium configured to store computer readable instructions, which when executed by a controller of a robotic system, cause the controller to perform a method comprising:

receiving, from a hand tracking system, a plurality of locations of a hand;

determining, by the controller, if the hand is in a first hand pose based on the plurality of locations;

in response to determining that the hand is in the first hand pose, switching, by the controller, to detecting hand trajectories while the robotic system is operating in a first mode of operation of a plurality of modes of operation; and

while detecting the hand trajectories:

detecting, based on hand tracking information from the hand tracking system, that the hand has performed a first hand trajectory of a plurality of known hand trajectories; and

in response to detecting the first hand trajectory, changing a mode of operation of the robotic system to a second mode of operation of the plurality of modes of operation, wherein the second mode of operation is selected from the group consisting of: a mode of operation for controlling an associated instrument to follow motions of an input device, a mode of operation decoupling control of the associated instrument from the input device, a mode of operation for controlling an imaging device, a mode of operation for swapping the input device from control of the associated instrument to control of a second instrument, and a mode of operation for toggling display of an auxiliary video window on a display.

18. The non-transitory tangible medium of claim 17 , wherein:

the hand tracking information comprises a sequence of unit velocities of the hand; and

detecting the first hand trajectory comprises comparing the sequence of unit velocities to sequences of unit velocities associated with each of the plurality of known hand trajectories.

19. The non-transitory tangible medium of claim 17 , wherein determining that the hand is in the first hand pose comprises matching the plurality of locations to the first hand pose using a classifier.

20. The non-transitory tangible medium of claim 17 , wherein changing the mode of operation comprises:

enabling a display mode;

disabling a display mode; or

switching to a new display mode.