Mobile virtual reality system for surgical robotic systems
Mobile virtual reality system for simulation, training or demonstration of a surgical robotic system can include a virtual reality processor. The processor can generate a virtual surgical robot and render the virtual surgical robot on a display. The virtual surgical robot can include a virtual surgical tool. A handheld user input device (UID) can sense a hand input from a hand. A foot input device can sense a foot input from a foot. The virtual reality processor can be configured to control a movement or action of the virtual surgical robot based on the hand input, and change which of the virtual surgical instruments is controlled by the one or more handheld UIDs based on the foot input. Other embodiments and aspects are disclosed and claimed.
1 . A method for providing an immersive mobile interface for simulation and teleoperation of a surgical robotic system, the method comprising:
receiving, by a head-worn device that is being worn by a user, a first video stream that comprises a virtual surgical robot within a virtual environment;
receiving, by the head-worn device, a second video stream that comprises a view of a foot or a hand of the user as the foot or the hand operate a user control for manipulating the virtual surgical robot; and
displaying, on a display of the head-worn device, a first view section that is above a second view section that is separate from the first view section, wherein the first view section comprises the first video stream and the second view section that comprises the second video stream.
2 . The method of claim 1 further comprising capturing, using a camera of the head-worn device, image data that includes the view of the foot or the hand of the user, wherein the second video stream is based on the image data.
3 . The method of claim 2 , wherein receiving the second video stream comprises producing the second video stream as a virtual representation of the view of the foot or the hand of the user by using the image data.
4 . The method of claim 1 , wherein the second video stream further comprises the user control operable by the foot or the hand of the user, or a virtualization of the user control.
5 . The method of claim 1 further comprising:
receiving, by the head-worn device, tracking data from a tracking system that is integrated into the head-worn device; and
manipulating, by the head-worn device and responsive to receiving the tracking data, the virtual surgical robot.
6 . The method of claim 1 , wherein the first view section is larger than the second view section within the display.
7 . The method of claim 4 , wherein the second video stream comprises the virtualization of the user control that is rendered with a physical view of the foot or the hand of the user that is operating the user control.
8 . A head-worn device comprising:
a display;
one or more processors; and
memory having stored therein instructions which when executed by the one or more processors causes the head-worn device to receive, during a simulation or a teleoperation of a surgical robotic operating procedure and while the head-worn device is worn by a user, a video stream that comprises a virtual surgical robot within a virtual environment,
wherein the display includes a first view section that is above a second view section that is sperate from the first view section, wherein the first view section comprises a rendering of the video stream and the second view section comprises a physical view of a foot or a hand of the user of the head-worn device as the foot or the hand of the user operate a user control for manipulating the virtual surgical robot.
9 . The head-worn device of claim 8 , wherein the second view section comprises an unobstructed opening of the display that is towards a nose of the user while the display is located over eyes of the user while the head-worn device is worn by the user to provide a line of sight between the eyes of the user and the foot or the hand of the user.
10 . The head-worn device of claim 8 , wherein the second view section is at a bottom portion of the display and further comprises a physical view of the user control operable by the foot or the hand of the user.
11 . The head-worn device of claim 8 , wherein the video stream is a first video stream, wherein the memory has further instructions to:
receive, during the simulation or the teleoperation of the surgical robotic operating procedure and while the head-worn device is worn by the user, a second video stream that comprises the physical view of the foot or the hand of the user of the head-worn device; and
display, on the display, the rendering of the first video stream within the first view section and a rendering of the second video stream in the second view section.
12 . The head-worn device of claim 11 further comprises a camera, wherein the instructions to receive the second video stream comprises instructions to capture, using the camera, image data that includes the physical view of the foot or the hand of the user of the head-worn device.
13 . The head-worn device of claim 12 , wherein the instructions to receive the second video stream further comprises instructions to produce the second video stream as a virtual representation of the foot or the hand of the user using the image data.
14 . The head-worn device of claim 13 , wherein the image data further comprises the user control operable by the foot or the hand of the user, wherein the second video stream comprises the user control or a virtualization of the user control.
15 . A processor of a head-worn device, wherein the processor is configured to:
receive, during a simulation or a teleoperation of a surgical robotic operating procedure and while the head-worn device is worn by a user, a first video stream that comprises a virtual surgical robot within a virtual environment and a second video stream that comprises a foot or a hand of the user as the foot or the hand of the user operate a user control for manipulating the virtual surgical robot; and
render, onto a display of the head-worn device, a first view section that is above a second view section that is separate from the first view section, wherein the first view section includes the first video stream and the second view section that includes the second video stream.
16 . The processor of claim 15 , wherein the second video stream comprises image data captured by a camera of the head-worn device that includes a physical view of the foot or the hand of the user, while the head-worn device is worn on a head of the user.
17 . The processor of claim 15 receives the second video stream by producing the second video stream that includes a virtual representation of the foot or the hand of the user based on sensor data captured by one or more sensors within a physical environment of the user.
18 . The processor of claim 16 , wherein the second video stream further comprises the physical view of the foot or the hand of the user and a virtualization of the user control that is based on the image data.
19 . The processor of claim 15 is further configured to:
receive tracking data from a tracking system that is integrated into the head-worn device; and
manipulate, responsive to receiving the tracking data, the virtual surgical robot.
20 . The processor of claim 15 , wherein the first view section is larger than the second view section, and the second view section is at a bottom of the display.