Assistance device and method for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument

1. An assistance device for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument, said assistance device comprising:

a camera for generating a video signal that contains a sequence of image frames;

a display unit for displaying the sequence of image frames on the basis of the video signal;

an image processing unit having an instrument detection module for detecting at least one target structure that represents the instrument being used in a frame of the sequence of image frames by identifying a predetermined distinguishing feature and extracting position information that indicates a position of the target structure in the frame;

a manipulator, which is coupled to the camera and can be actuated via a control signal to move the camera; and

a manipulator controller for generating the control signal from the position information and for actuating the manipulator via the control signal;

wherein the instrument detection module identifies an image segment as the predetermined distinguishing feature, said image segment being characterized by a color saturation that is equal to or less than a predefined color saturation, and by an outline that delimits the image segment and has at least one rectilinear section,

wherein the image processing unit includes a segmentation module, which generates a plurality of binary images on the basis of the frame of the sequence of image frames,

wherein the instrument detection module identifies mutually corresponding rectilinear sections in the binary images and combines these sections to form a composite line segment as the image segment which represents an edge line of the instrument, and

wherein the segmentation module generates the plurality of binary images on the basis of a grayscale image generated by a preprocessing module of the image processing unit based on the frame of the sequence of image frames, and applies different threshold values to the grayscale image for the generation of said binary images.

2. The assistance device according to claim 1 , wherein

the preprocessing module generates the at least one grayscale image on the basis of the frame of the sequence of image frames, the pixels of which are each assigned a grayscale value that represents the color saturation of the corresponding pixel of the frame; and

on the basis of the grayscale image, the segmentation module generates the at least one binary image, the binary pixels of which are assigned a first binary value if the associated grayscale values are equal to or less than a threshold value that corresponds to the predefined color saturation, and the binary pixels of which are assigned a second binary value if the associated grayscale values are greater than the threshold value.

3. The assistance device according to claim 1 , wherein the instrument detection module combines a plurality of collinearly spaced rectilinear sections of the outline to form a continuous edge line that represents an edge of the instrument.

4. The assistance device according to claim 3 , wherein for detection of the target structure, the instrument detection module pairs two edge lines that are arranged parallel to one another in each case.

5. The assistance device according to claim 4 , wherein the instrument detection module identifies an instrument tip on the basis of the two paired edge lines.

6. The assistance device according to claim 1 , wherein the image processing unit includes a tracking module, which tracks the target structure detected by the instrument detection module over multiple successive frames; and

the manipulator controller generates the control signal related to the frame of the sequence of image frames, using the position information about the target structure, for the purpose of actuating the manipulator only if said target structure has already been tracked by the tracking module over multiple successive frames.

7. The assistance device according to claim 6 , wherein the tracking module assigns a tracker to a target structure that is detected for the first time in the frame of the sequence of image frames by the instrument detection module, and uses the tracker to track said target structure that is detected in the subsequent frames.

8. The assistance device according to claim 1 , wherein the image processing unit includes a flow module, which detects an optical flow of the image sequence, which represents movement information contained in the image sequence.

9. The assistance device according to claim 6 , wherein

the tracking module comprises a first submodule, located upstream of the instrument detection module, and a second submodule, located downstream of the instrument detection module;

the first submodule factors in the optical flow detected by the flow module to make a prediction regarding the position information about the tracker for the next frame, which has not yet been processed by the instrument detection module;

and for said next frame, which has been processed by the instrument detection module, the second submodule verifies the prediction made by the first submodule based on the position information for the tracker detected by the instrument detection module.

10. The assistance device according to claim 2 , wherein the preprocessing module is designed to perform a white balance adjustment of the frame of the sequence of image frames.

11. The assistance device according to claim 10 , wherein the image processing unit includes a parameter optimization module, which processes the frames asynchronously to the preprocessing module and from them generates actuation information, which specifies whether or not the preprocessing module should perform the white balance adjustment.

12. The assistance device according to claim 11 , wherein the parameter optimization module predefines the threshold values for the segmentation module on the basis of the asynchronous processing of the frames.

13. A method for providing imaging support to an operating surgeon during a surgical procedure involving at least one medical instrument, comprising:

generating, with a camera, a video signal that includes a sequence of image frames;

displaying the image sequence on a display unit on the basis of the video signal;

detecting at least one target structure that represents the instrument being used in a frame of the sequence of image frames by identifying a predefined distinguishing feature, and position information that indicates a position of the target structure in the frame;

generating a control signal from the position information, and

actuating a manipulator coupled to the camera via the control signal to move the camera;

wherein an image segment is identified as the predefined distinguishing feature, said image segment being characterized by a color saturation that is equal to or less than a predefined color saturation, and by an outline that delimits the image segment and has at least one rectilinear section,

wherein the image segment is identified by generating a plurality of binary images on the basis of the frame of the sequence of image frames, identifying mutually corresponding rectilinear sections in the binary images, and combining these sections to form a composite line segment as the image segment which represents an edge line of the instrument, and

wherein the plurality of binary images are generated on the basis of a grayscale image generated by a preprocessing module based on the frame of the sequence of image frames by applying different threshold values to the grayscale image for the generation of said binary images.