Collaborative mixed-reality system for immersive surgical telementoring
Systems and methods utilize mixed reality technologies to enable remotely located expert surgeons to instruct novice surgeons as if they were together in the same operating room. An example method may involve: (1) displaying, on a virtual reality (VR) display worn by a first person located in a second geographic location, a 3D virtual reconstruction of a patient located in a first geographic location; (2) determining a spatial relationship between a hand gesture performed by the first person and the 3D virtual reconstruction of the patient; and (3) displaying, on an augmented reality (AR) display worn by a second person located in the first geographic location, a 3D avatar of the hand gesture made to appear in a spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand gesture and the 3D virtual reconstruction of the patient.
1 . A method comprising:
obtaining, from a depth camera located in a first geographic location, 3D images of a surgical subject located in the first geographic location;
displaying, on a virtual reality (VR) display worn by a first person located in a second geographic location, a 3D virtual reconstruction of the surgical subject based on the obtained 3D images of the surgical subject;
determining a spatial relationship between a hand gesture performed by the first person and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display;
displaying, on an augmented reality (AR) display worn by a second person located in the first geographic location, a 3D avatar of the hand gesture made to appear in a spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand gesture and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display; and
responsive to manual inputs provided by the first person, modifying display parameters of the AR display worn by the second person, wherein modifying the display parameters of the AR display worn by the second person comprises at least one of:
changing visibility of the 3D avatar of the hand gesture, or causing a supplemental instructional video to be displayed and played on the AR display in addition to the 3D avatar of the hand gesture.
2 . The method of claim 1 , wherein the first person is providing surgical instruction to the second person.
3 . The method of claim 1 , wherein determining the spatial relationship between the hand gesture and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display comprises:
obtaining 3D position information and 3D orientation information related to the hand gesture and a physical surface onto which the 3D virtual reconstruction of the surgical subject is made to appear on the VR display.
4 . The method of claim 3 , wherein obtaining the 3D position information and 3D orientation information related to the hand gesture and the physical surface onto which the 3D virtual reconstruction of the surgical subject is made to appear on the VR display comprises using at least one of a motion capture system associated with the second geographic location and a depth camera located at the second geographic location.
5 . The method of claim 4 , wherein the motion capture system associated with the second geographic location comprises a motion capture camera located at the second geographic location and tracker markers located on the gesturing hand of the first person and the physical surface onto which 3D virtual reconstruction of the surgical subject is made to appear.
6 . The method of claim 5 , wherein:
the hand gesture comprises an annotation performed proximate to an anatomical region of the 3D virtual reconstruction of the surgical subject.
7 . The method of claim 6 , wherein:
the annotation is performed using a pen controller for making virtual written annotations; and
the 3D position information and 3D orientation information related to the annotation is obtained using the motion capture camera located at the second geographic location and a tracker marker attached to the pen controller.
8 . The method of claim 6 , wherein displaying the 3D avatar of the annotation to appear in the spatial relationship to the surgical subject that mirrors the determined spatial relationship between the annotation and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display comprises:
displaying, on the AR display worn by the second person located in the first geographic location, a 3D virtual written annotation made to appear proximate to an anatomical region of the surgical subject corresponding to the anatomical region of the 3D virtual reconstruction of the surgical subject.
9 . A collaborative mixed reality system comprising:
a depth camera that obtains 3D images of a surgical subject located in a first geographic location;
a VR display worn by a first person located in a second geographic location that displays a 3D virtual reconstruction of the surgical subject based on the obtained 3D images of the surgical subject;
a motion capture camera located in the second geographic location that obtains 3D position information and 3D orientation information for a hand of the first person and a surface onto which the 3D virtual reconstruction of the surgical subject is made to appear on the VR display;
one or more processing resources and non-transitory computer-readable medium, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resources cause the collaborative mixed reality system to:
use the 3D position information and 3D orientation information obtained by the motion capture camera located in the second geographic location to determine a spatial relationship between the hand of the first person and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display; and
an AR display worn by a second person located in the first geographic location that:
displays a 3D avatar of the hand made to appear in a spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display; and
a control panel located at the second geographic location configured to permit the first person to modify display parameters of the AR display by at least one of: changing visibility of the 3D avatar of the hand, or causing a supplemental instructional video to be displayed and played on the AR display in addition to the 3D avatar of the hand.
10 . The collaborative mixed reality system of claim 9 , further comprising:
a motion tracking system associated with the first geographic location, the motion tracking system associated with the first geographic location comprising a motion capture camera located at the first geographic location and tracker markers associated with the AR display and the depth camera respectively.
11 . The collaborative mixed reality system of claim 10 , wherein the instructions executed by the one or more processing resources further cause the collaborative mixed reality system to:
calibrate a coordinate system of the AR display in order to display the 3D avatar of the hand to appear in the spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display.
12 . The collaborative mixed reality system of claim 11 , wherein calibrating the coordinate system of the AR display in order to display the 3D avatar of the hand to appear in the spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display comprises:
performing a first calibration to calibrate the coordinate system of the AR display with a coordinate system of the depth camera; and
performing a second calibration to calibrate the first calibrated coordinate systems with a coordinate system of the VR display.
13 . The collaborative mixed reality system of claim 12 , wherein calibrating the coordinate system of the AR display with the coordinate system of the depth camera comprises:
using the motion tracking system associated with the first geographic location to determine a spatial relationship between the AR display and the depth camera;
synchronizing the AR display and the depth camera using a calibration marker in view of the AR display and the depth camera; and
calibrating the coordinate system of the AR display with the coordinate system of the depth camera based on the determined spatial relationship between the AR display and the depth camera and the synchronization between the AR display and the depth camera.
14 . The collaborative mixed reality system of claim 12 , wherein calibrating the first calibrated coordinate systems with the coordinate system of the VR display comprises:
using the motion capture camera located in the second geographic location to determine a spatial relationship between the VR display and the surface onto which the 3D virtual reconstruction of the surgical subject is made to appear on the VR display.
15 . The collaborative mixed reality system of claim 9 , wherein the collaborative mixed reality system further comprises:
a pen controller for making virtual written annotations and a tracker marker located on the pen controller.
16 . The collaborative mixed reality system of claim 15 , wherein the motion capture camera located in the second geographic location further obtains 3D position information and 3D orientation information for the pen controller using the tracker marker located on the pen controller.
17 . The collaborative mixed reality system of claim 16 , wherein:
the instructions executed by the one or more processing resources further cause the collaborative mixed reality system to determine, based on the 3D position information and 3D orientation information obtained by the motion capture camera located in the second geographic location, a spatial relationship between the pen controller and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display; and
the AR display worn by the second person located in the first geographic location further displays a 3D avatar of the pen controller made to appear in a spatial relationship to the surgical subject that mirrors the determined spatial relationship between the pen controller and 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display.
18 . The collaborative mixed reality system of claim 17 , wherein the AR display further displays the 3D avatar of the pen controller making a 3D virtual written annotation.
19 . A system comprising:
one or more processing resources; and
non-transitory computer-readable medium, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resources cause the system to:
obtain 3D images of a surgical subject located in a first geographic location, display, on a VR display worn by a first person located in a second geographic location, a 3D virtual reconstruction of the surgical subject based on the obtained 3D images of the surgical subject, determine a spatial relationship between a hand of the first person and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display, display, on an AR display worn by a second person located in the first geographic location, a 3D avatar of the hand of the first person made to appear in a spatial relationship to the surgical subject that mirrors the determined spatial relationship between the hand of the first person and the 3D virtual reconstruction of the surgical subject as the 3D virtual reconstruction of the surgical subject is made to appear on the VR display, and responsive to manual inputs provided by the first person, modify display parameters of the AR display worn by the second person, wherein modifying the display parameters of the AR display worn by the second person comprises at least one of:
changing visibility of the 3D avatar of the hand, or causing a supplemental instructional video to be displayed and played on the AR display in addition to the 3D avatar of the hand.
20 . The system of claim 19 , wherein the first person is providing surgical instruction to the second person.