Anthro-centric multisensory interface for sensory augmentation of telesurgery

1. A method for allowing a surgeon to control a robotic surgical apparatus including an end effector, comprising:

a. providing control software running on a computer;

b. providing a gesture capture system, said gesture capture system capturing present location information for at least said surgeon s fingers and forearms;

c. said gesture capture system providing said present location information for at least said surgeon's fingers and forearms to said control software;

d. said control software controlling operation of said end effector;

e. said control software using said present location information for at least said surgeon's fingers and forearms to control said operation of said end effector;

f. providing a depth-sensing camera system, said camera system providing data sufficient to generate a three dimensional software model of structures within said camera system's field of view;

g. using said camera system data to create a three dimensional software model of said structures within said camera system's field of view;

h. providing a video display, including a pan function;

i. displaying at least a portion of said three dimensional software model of said structures on said video display;

j. providing a head motion capture system, said head motion capture system capturing present head location information fur said surgeon's head; and

k. controlling said pan function on said video display using said head location for information for said surgeon's head.

2. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comprising:

a. providing an apparent viewpoint for said depth-sensing camera system; and

b. providing a simulated parallax function as part of said video display, wherein when said pan function is activated, objects that are closer to said apparent viewpoint move a greater distance on said display that objects that are farther away.

3. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comrising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon's head.

4. A method for allowing a surgeon to control a robotic surgical apparatus as recite in claim 2 , further comprising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon's head.

5. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comprising:

a. panning said video display to the left when said surgeon's head moves left;

b. panning said video display to the right when said surgeon's head moves right;

c. panning said video display up when said surgeon's head moves down; and

d. panning said video display down when said surgeon's head moves up.

6. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 2 , further comprising:

a. panning said video display to the right when said surgeon's head moves left;

b. panning said video display to the left when said surgeon's head moves right;

c. panning said video display up when said surgeon's head moves down;

d. panning, said video display down when said surgeon's head moves up;

e. providing a zoom function in said video display; and

f. controlling said zoom function on said video display using said head location capture information for said surgeon's head.

7. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comprising:

a. providing a zoom function in said video display; and

b. initially setting said zoom function so that said end effector's display on said monitor is scaled to approximately match a forearm and a hand of said surgeon.

8. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comprising providing specific defined positions for said surgeon's fingers which are interpreted by said control software as specific defined commands.

9. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 8 , wherein a clenched fist is defined as a command to lock a selected effector in position.

10. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , further comprising:

a. allowing said surgeon to control a second robotic surgical apparatus;

b. wherein said robotic surgical apparatus is associated with a first patient and said second robotic surgical apparatus is associated with a second patient; and

c. allowing said surgeon to toggle control between said robotic surgical apparatus and said second robotic surgical apparatus.

11. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 1 , thriller comprising:

a. providing a sensory feedback display to said surgeon; and

b. using said sensory feedback display to display a state of said end effector.

12. method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11 , wherein said sensory feedback display is an electro-tactile array.

13. A method for allowing, a surgeon to control a robotic surgical apparatus as recited in claim 11 , wherein said sensory feedback display is a vibro-tactile torso vest.

14. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11 , wherein said state or said end effector is a position of said end effector.

15. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 11 , wherein;

a. said end effector has a movable portion; and

b. said state of said end effector is a position of said movable portion.

16. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 15 , wherein:

a. said end effector is able to sense a force feedback for said movable portion; and

b. said sate of said end effector is an amount of force feedback for said movable portion.

17. A method for allowing a surgeon to control a robotic surgical apparatus including an end effector, comprising:

a. providing control software running on a computer;

b. providing a gesture capture system, said gesture capture system capturing present location information for at least said surgeon's fingers and at least one wrist;

c. said gesture capture system providing said present location information for at least said surgeon's fingers and said at least one wrist to said control software;

d. providing specific defined Positions for said surgeon's fingers which are interpreted by said control software as specificdefined commands;

e. wherein the posititin of said surgeon's fingers and said at least one wrist are used by said control software to control the veration of said end effector;

f. providing a depth-sensing camera system, said camera system providing data sufficient to generate a three dimensional software model of structures within said camera system's field of view;

g. using said camera system data to create a three dimensional software model of said structures within said camera system's field of view;

h. providing a video display;

i. displaying at least a portion of said three dimensional software model of said structures on said video display;

j. providing a pan function for said video display;

k. providing a head motion capture system, said head motion capture system capturing present head location information for said surgeon's head; and

l. controlling said pan function on said video display using said head location information for said surgeon's head.

18. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. providing an apparent viewpoint for said depth-sensing camera system; and

b. providing a simulated parallax function as part of said video display, wherein when said pan function is activated, objects that are closer to said apparent viewpoint move a greater distance on said display that objects that are farther away.

19. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. providing a zoom function in said video display; and

b. controlling said zoom function on said video display using said head location capture information for said surgeon's head.

20. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. panning said video display to the right when said surgeon's head moves left;

b. panning said video display to the left when said surgeon's head moves right;

c. panning said video display up when said surgeons head moves down; and

d. panning said video display down when said surgeon's head moves up.

21. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim l 7 , further comprising:

a. panning said video display to the right when said surgeon's head moves left;

b. panning said video display to the left when said surgeon's head moves right;

c. panning said video display up when said surgeons head moves down; and

d. panning said video display down when said surgeon's head moves up;

e. providing a zoom function in said video display; and

f. controlling said zoom function on said video display using said head location capture information for said surgeon's head.

22. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. providing a zoom function in said video display; and

b. initially setting said zoom function so that said end effector's display on said monitor is scaled to approximately match a wrist and a hand of said surgeon.

23. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. providing a zoom function in said video display;

b. initially setting said zoom function so that said end effector's display on said monitor is scaled to approximately match a wrist and a band of said surgeon.

24. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , wherein a clenched first is defined as a command to lock a selected effector in position.

25. A method for allowing as surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. allowing said surgeon to control a second robotic surgical apparatus;

b. wherein said robotic surgical apparatus is associated with a first patient and said second robotic surgical apparatus is associated with is a second patient; and

c. allowing said surgeon to toggle control between said robotic surgical apparatus and said second robotic surgical apparatus.

26. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 17 , further comprising:

a. providing a sensory feedback display to said surgeon; and

b. using said sensory feedback display to display a state of said end effector.

27. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26 , wherein said sensory feedback display is an electro-tactile array.

28. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26 , wherein said sensory feedback display is a vibro-tactile torso vest.

29. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26 , wherein said state of said end effector is a position of said end effector.

30. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 26 , wherein:

a. said end effector has a movable portion; and

b. said state of said end effector is a position of said movable portion.

31. A method for allowing a surgeon to control a robotic surgical apparatus as recited in claim 30 , wherein:

a. said end effector is able to sense a force feedback for said movable portion; and

b. said sate of said end effector is an amount of force feedback for said movable portion.