System, method and software program for aiding in positioning of objects in a surgical environment
Systems, methods and computer-program products are provided for aiding in positioning between a surgical object and a machine vision camera in a surgical environment. The machine vision camera senses a position and orientation of a surgical object in its field of view. Controller(s) virtually define a zone within the field of view that indicates a range of acceptable positions for the surgical object relative to a position of the machine vision camera. The controller(s) obtain an acceptable orientation range for the surgical object relative to an orientation of the machine vision camera. The controller(s) execute a GUI to simultaneously present, on the display device, representations of: the field of view; the zone within the field of view; the position of the surgical object relative to the zone; and a current angle difference between the surgical object and the machine vision camera.
1 . A surgical system comprising:
a display device;
a machine vision camera having a field of view and being configured to sense a position and orientation of a surgical object in the field of view; and
one or more controllers coupled to the machine vision camera and configured to:
virtually define a zone within the field of view, the zone being positioned at a location that is static relative to the field of view such that the zone is located independent of the sensed position of the surgical object, and the zone indicating a range of acceptable positions for the surgical object relative to a position of the machine vision camera;
obtain an acceptable orientation range for the surgical object relative to an orientation of the machine vision camera;
obtain the position and orientation of the surgical object from the machine vision camera; and
execute a graphical user interface configured to aid in positioning the machine vision camera and the surgical object relative to one another in a surgical environment, wherein the graphical user interface is configured to simultaneously present, on the display device, representations of:
the field of view;
the zone within the field of view at the location;
the position of the surgical object relative to the zone; and
a current angle difference between the surgical object and the machine vision camera.
2 . The surgical system of claim 1 , wherein the machine vision camera senses the position and the orientation of the surgical object by being configured to utilize: depth map generation, segmentation, edge detection, color analysis, pixel analysis, and/or pattern recognition.
3 . The surgical system of claim 1 , wherein the machine vision camera senses the position and the orientation of the surgical object without a tracker being attached to the surgical object.
4 . The surgical system of claim 1 , wherein the acceptable orientation range is defined as an angle difference between the surgical object and the machine vision camera being less than or equal to 20 degrees.
5 . The surgical system of claim 1 , wherein the one or more controllers are configured to execute the graphical user interface to further present, on the display device:
a first visual feedback in response to the current angle difference being within the acceptable orientation range; and
a second visual feedback in response to the current angle difference being outside of the acceptable orientation range.
6 . The surgical system of claim 1 , wherein, in response to the surgical object being within the zone and the current angle difference being within the acceptable orientation range, the one or more controllers are configured to automatically accept the relative positioning between the machine vision camera and the surgical object.
7 . The surgical system of claim 6 , wherein the one or more controllers are configured to automatically advance the graphical user interface to a subsequent screen in response to automatically accepting the relative positioning between the machine vision camera and the surgical object.
8 . The surgical system of claim 1 , wherein the zone is predefined based on a pre-operative surgical plan including data related to planned positioning of the machine vision camera and the surgical object.
9 . The surgical system of claim 1 , wherein the zone is predefined based on a computer simulation including data related to the machine vision camera and the surgical object.
10 . The surgical system of claim 1 , wherein the zone is defined in response to the one or more controllers performing an initial evaluation of the obtained positions of the surgical object.
11 . The surgical system of claim 1 , wherein the one or more controllers execute the graphical user interface to present, on the display device, the representation of the position of the surgical object as a shape object.
12 . The surgical system of claim 1 , wherein the one or more controllers execute the graphical user interface to present, on the display device, the representation of the field of view and the position of the surgical object based on a real-world video stream of the surgical environment obtained by the machine vision camera.
13 . The surgical system of claim 1 , wherein the one or more controllers execute the graphical user interface to present, on the display device, a numerical indication of a current distance of the surgical object relative to the machine vision camera.
14 . The surgical system of claim 13 , further comprising a laser range finder and wherein the one or more controllers are configured to utilize the laser range finder to determine the current distance of the surgical object relative to the machine vision camera.
15 . The surgical system of claim 1 , wherein the one or more controllers determine the current angle difference between the surgical object and the machine vision camera by being configured to compare a direction perpendicular to a primary operational surface of the surgical object relative to a direction perpendicular to a surface of the machine vision camera.
16 . The surgical system of claim 1 , wherein:
the display device is incorporated in a head-mounted device that is to be worn by a user to aid the user in positioning the machine vision camera and the surgical object relative to one another; and
the one or more controllers are configured to execute the graphical user interface on the display device of the head-mounted device using augmented reality or mixed reality by mixing real-world views or real-world video of the surgical environment with computer-generated images of the representations of one or more of: field of view, the zone, and/or the position of the surgical object.
17 . A non-transitory computer-readable medium for a surgical system including a display device and a machine vision camera having a field of view and being configured to sense a position and orientation of a surgical object in the field of view, the non-transitory computer-readable medium comprising instructions, which when executed by one or more processors, are configured to:
virtually define a zone within the field of view, the zone being positioned at a location that is static relative to the field of view such that the zone is located independent of the sensed position of the surgical object, and the zone indicating a range of acceptable positions for the surgical object relative to a position of the machine vision camera;
obtain an acceptable orientation range for the surgical object relative to an orientation of the machine vision camera;
obtain the position and orientation of the surgical object from the machine vision camera; and
execute a graphical user interface configured to aid in positioning the machine vision camera and the surgical object relative to one another in a surgical environment, wherein the graphical user interface is configured to simultaneously present, on the display device, representations of:
the field of view;
the zone within the field of view at the location;
the position of the surgical object relative to the zone; and
a current angle difference between the surgical object and the machine vision camera.
18 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the one or more processors, are configured to execute the graphical user interface to further present, on the display device:
a first visual feedback in response to the current angle difference being within the acceptable orientation range; and
a second visual feedback in response to the current angle difference being outside of the acceptable orientation range.
19 . The non-transitory computer-readable medium of claim 17 , wherein, in response to the surgical object being within the zone and the current angle difference being within the acceptable orientation range, the instructions, when executed by the one or more processors, are configured to automatically accept the relative positioning between the machine vision camera and the surgical object.
20 . The non-transitory computer-readable medium of claim 19 , wherein the instructions, when executed by the one or more processors, are configured to automatically advance the graphical user interface to a subsequent screen in response to automatically accepting the relative positioning between the machine vision camera and the surgical object.
21 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the one or more processors, are configured to execute the graphical user interface to present, on the display device, the representation of the position of the surgical object as a shape object.
22 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the one or more processors, are configured to execute the graphical user interface to present, on the display device, the representation of the field of view and the position of the surgical object based on a real-world video stream of the surgical environment obtained by the machine vision camera.
23 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the one or more processors, are configured to execute the graphical user interface to present, on the display device, a numerical indication of a current distance of the surgical object relative to the machine vision camera.
24 . The non-transitory computer-readable medium of claim 17 , wherein the instructions, when executed by the one or more processors, determine the current angle difference between the surgical object and the machine vision camera by being configured to compare a direction perpendicular to a primary operational surface of the surgical object relative to a direction perpendicular to a surface of the machine vision camera.
25 . A method of operating a surgical system, the surgical system including a display device, a machine vision camera having a field of view and being configured to sense a position and orientation of a surgical object in the field of view, and one or more controllers coupled to the machine vision camera, the method comprising the one or more controllers performing the following steps:
virtually defining a zone within the field of view, the zone being positioned at a location that is static relative to the field of view such that the zone is located independent of the sensed position of the surgical object, and the zone indicating a range of acceptable positions for the surgical object relative to a position of the machine vision camera;
obtaining an acceptable orientation range for the surgical object relative to an orientation of the machine vision camera;
obtaining the position and orientation of the surgical object from the machine vision camera; and
executing a graphical user interface for aiding in positioning the machine vision camera and the surgical object relative to one another in a surgical environment, by simultaneously presenting, on the display device, representations of:
the field of view;
the zone within the field of view at the location;
the position of the surgical object relative to the zone; and
a current angle difference between the surgical object and the machine vision camera.