Augmented surgical reality environment for a robotic surgical system
The present disclosure is directed to robotic surgical system that includes an operating console configured to operate at least one robotic arm and at least one drive motor configured to receive an input from the operating console and control the at least one robotic arm based on the input from the operating console. A surgical instrument is coupled to the at least one robotic arm. The surgical instrument is inserted into a patient and captures an image of a region of interest inside the patient during a surgical procedure. A controller receives the image and applies at least one image processing filter to identify at least one non-visible property of an object in the region of interest. A display displays the image to a user.
1. A robotic surgical system comprising:
a plurality of robotic arms;
an image capture device attached to a surgical assembly of a first of the robotic arms and configured to be inserted into a patient and capture an image of a region of interest inside the patient during a surgical procedure;
a surgical instrument attached to a surgical assembly of a second of the robotic arms configured to be inserted into a patient and manipulate tissue inside the patient during the surgical procedure;
an input device configured to manipulate at least one of the image capture device attached to the first robotic arm and the surgical instrument attached to the second robotic arm during the surgical procedure;
an image processing filter configured to filter image data from the captured image at a predetermined spatial frequency to identify a property of an object imperceptible at the predetermined spatial frequency, the image processing filter including:
a spatial decomposition filter configured to decompose the image into a plurality of spatial frequency bands;
a temporal filter configured to be applied to the plurality of spatial frequency bands to generate a plurality of temporally filtered bands;
an adder configured to add each band in the plurality of spatial frequency bands to a corresponding band in the plurality of temporally filtered bands to generate a plurality of augmented bands; and
a reconstruction filter configured to generate an augmented image by collapsing the plurality of augmented bands;
a controller configured to enhance a perceptibility of the identified property of the object and output the enhanced perceptibility of the identified property; and
a display configured to display the captured image during the surgical procedure.
2. The robotic surgical system of claim 1 , wherein the controller includes a graphics processor configured to enhance a visible representation of the identified property and output the enhanced visible representation to the display to be displayed.
3. The robotic surgical system of claim 2 , wherein the controller is configured to amplify the filtered image data to enhance the visible representation of the identified property.
4. The robotic surgical system of claim 2 , wherein the controller is configured to transition at least a portion the filtered image data having a non-visible frequency to a frequency in a visible light frequency range to enhance the visible representation of the identified property.
5. The robotic surgical system of claim 1 wherein the controller includes a haptic processor configured to enhance a force-feedback representation of the identified property and output the enhanced force-feedback representation to a haptic device on the input device configured to provide haptic feedback to the user.
6. The robotic surgical system of claim 5 , wherein the controller is configured to amplify the filtered image data to enhance the force-feedback representation of the identified property.
7. The robotic surgical system of claim 1 , further comprising a control device configured to:
identify the surgical instrument attached to the second robotic arm; and
select an image processing filter from a plurality of image processing filters based on the identification of the surgical instrument.
8. The robotic surgical system of claim 7 , wherein the control device is further configured to:
determine whether the surgical instrument attached to the second robotic arm is within a field of view of the image capture device during the surgical procedure; and
select the image processing filter if the surgical instrument is within the field of view of the image capture device.
9. The robotic surgical system of claim 8 , wherein:
the plurality of arms includes a third robotic arm having a surgical assembly to which a surgical instrument is attached;
the control device is further configured to identify the surgical instrument attached to the third arm;
the control device is further configured to determine which of the surgical instruments attached to the second and the third robotic arms appear within the field of view of the image capture device; and
the control device is further configured to select the image processing filter based on the determination.
10. The robotic surgical system of claim 7 , wherein the control device is further configured to:
identify a task being performed by the identified surgical instrument; and
select the image processing filter based on the task being performed during the surgical procedure.
11. The robotic surgical system of claim 7 , wherein the control device is further configured to:
identify an initial task being performed by the identified surgical instrument during the surgical procedure;
initially select and apply the image processing filter to initially captured image data based on the initial task;
identify a subsequent task being performed by the identified surgical instrument during the surgical procedure; and
select and apply a different image processing filter to subsequently captured image data based on the subsequent task.
12. The robotic surgical system of claim 11 , wherein the control device is further configured to associate movement of the surgical instrument at a rate less than a threshold as the initial task and associate movement of the surgical instrument at a rate greater than the threshold with the subsequent task.
13. The robotic surgical system of claim 11 , wherein the control device is further configured to associate applied to movement of the surgical instrument at a first scaling rate with the initial task and associate movement of the surgical instrument at a different scaling factor rate with the subsequent task.
14. The robotic surgical system of claim 11 , wherein the control device is further configured to associate an actuation of the surgical instrument within a first period with the initial task and associate a non-actuation of the surgical instrument within a second period with the subsequent task.
15. A method comprising:
capturing an image of a region of interest inside a patient during a surgical procedure with an image capture device attached to a surgical assembly of a first of a plurality of robotic arms of a surgical robotic system;
manipulating a surgical instrument inside the patient during the surgical procedure after the surgical instrument is attached to a surgical assembly of a second of the plurality of robotic arms during the surgical procedure responsive to a user manipulating an input device of the surgical robotic system;
filtering image data from the captured image at a predetermined spatial frequency to identify a property of an object imperceptible at the predetermined spatial frequency;
enhancing a perceptibility of the identified property of the object and output the enhanced perceptibility of the identified property; and
outputting the enhanced perceptibility to the user of the surgical robotic system; and
providing information to the user in the outputted enhanced perceptibility relating to the manipulation of the surgical instrument.
16. The method of claim 15 , wherein the additional information provided to the user is a visualization of a vessel seal effectiveness.
17. The method of claim 15 , wherein the additional information provided to the user is a visualization of abnormal tissue.
18. The method of claim 15 , wherein the enhanced perceptibility of the identified property of the object is a location of delicate tissue and the additional information provided to the user is a force-feedback applied at input device of the robotic surgical system responsive to the user moving the input device so as to manipulate the surgical instrument within a predetermined distance of the delicate tissue.
19. A non-transitory computer readable medium storing instructions that, when executed by a processing device, cause the processing device to:
filter image data at a predetermined spatial frequency to identify a property of an object imperceptible at the predetermined spatial frequency from an image of a region of interest inside a patient during a surgical procedure obtained from an image capture device attached to a surgical assembly of a first of a plurality of robotic arms of a surgical robotic system;
manipulate a surgical instrument inside the patient during the surgical procedure after the surgical instrument is attached to a surgical assembly of a second of the plurality of robotic arms during the surgical procedure responsive to a user manipulating an input device of the surgical robotic system;
enhance a perceptibility of the identified property of the object from the filtered image data;
output the enhanced perceptibility to a user of the surgical robotic system; and
provide additional information to the user in the outputted enhanced perceptibility relating to the manipulation of the surgical instrument.