Endoscopic surgery system, image processing apparatus, and image processing method
The present disclosure relates to an endoscopic surgery system, an image processing apparatus, and an image processing method that enable better endoscopic observation. Motion components at the time of imaging are estimated on the basis of an image captured by an endoscope, a degree of reliability indicating a degree that the motion components are motion components for the entire image is set, and a correction amount used when correction processing of correcting a blur is performed on the image is controlled according to the degree of reliability. For example, in a case where the motion components are single motion components, the degree of reliability is set to 1.0, and in a case where the motion components are a plurality of motion components, the degree of reliability is set to 0.0. The present technology can be applied to, for example, an endoscopic surgery system used in surgery using an endoscope.
1. An endoscopic surgery system, comprising:
an endoscope configured to capture an image;
a motion estimation unit configured to:
estimate one or more motion components at a time of the capture of the image, wherein the one or more motion components are estimated based on the captured; and
set, based on the estimated one or more motion components, a degree of reliability indicating that the estimated one or more motion components correspond to at least one of a motion of the endoscope or a motion of a surgical tool;
a correction amount control unit configured to control a correction amount based on the degree of reliability; and
a correction processing unit configured to correct, based on the controlled correction amount, a blur associated with the captured image.
2. The endoscopic surgery system according to claim 1 , wherein
the motion estimation unit is further configured to estimate whether the estimated one or more motion components include a single motion component or a plurality of motion components,
the single motion component has a single size and a single direction, and
the plurality of motion components has a plurality of different sizes and a plurality of different directions.
3. The endoscopic surgery system according to claim 2 , wherein the motion estimation unit is further configured to:
set a first value as the degree of reliability based on the estimation that the one or more motion components include the single motion component; and
set a second value as the degree of reliability based on the estimation that the one or more motion components include the plurality of motion components.
4. The endoscopic surgery system according to claim 3 , wherein
the first value is 1.0, and
the second value is 0.0.
5. The endoscopic surgery system according to claim 3 , further comprising an operation unit configured to:
acquire a plurality of local motion vectors indicating a local motion of the captured image;
determine, based on the plurality of local motion vectors, each of a norm of the plurality of local motion vectors and an angle of the plurality of local motion vectors; and
determine, based on the determined norm and the determined angle, each of:
a norm standard deviation of the plurality of local motion vectors, and
an angle standard deviation of the plurality of local motion vectors, wherein the motion estimation unit is further configured to:
obtain a threshold; and
estimate the one or more motion components of the image based on each of the obtained threshold, the determined norm standard deviation, and the determined angle standard deviation.
6. The endoscopic surgery system according to claim 5 , wherein the motion estimation unit is further configured to:
set, in a graph, a threshold curve that represents the threshold, wherein the graph includes the norm standard deviation as a vertical axis and the angle standard deviation as a horizontal axis;
estimate, in a case where the norm standard deviation and the angle standard deviation are in a first region of the graph, that the one or more motion components include the single motion component, wherein the first region is less than the threshold curve;
set the first value as the degree of reliability based on the estimation that the one or more motion components include the single motion component;
estimate, in a case where the norm standard deviation and the angle standard deviation are in a second region of the graph, that the one or more motion components include the plurality of motion components, wherein the second region is one of equal to or larger than the threshold curve; and
set the second value as the degree of reliability based on the estimation that the one or more motion components include the plurality of motion components.
7. The endoscopic surgery system according to claim 5 , wherein the motion estimation unit is further configured to:
set, in a graph, each of a first threshold curve and a second threshold curve, wherein
at least one of the first threshold curve and the second threshold curve represents the threshold, and
the graph includes the norm standard deviation as a vertical axis and the angle standard deviation as a horizontal axis;
estimate, in a case where the norm standard deviation and the angle standard deviation are in a first region, that the one or more motion components include the single motion component, wherein the first region is less than the first threshold curve;
set the first value as the degree of reliability based on the estimation that the one or more motion components include the single motion component:
set, in a case where the norm standard deviation and the angle standard deviation are in a second region of the graph, a third value, from the first value to the second value, as the degree of reliability, wherein the second region is each of:
one of equal to or larger than the first threshold curve, and
one of equal to or smaller than the second threshold curve;
estimate, in a case where the norm standard deviation and the angle standard deviation are in a third region of the graph, that the one or more motion components include the plurality of motion components, wherein the third region is larger than the second threshold curve; and
set the second value as the degree of reliability based on the estimation that the one or more motion components include the plurality of motion components.
8. The endoscopic surgery system according to claim 5 , further comprising:
a feature point extraction unit configured to extract a plurality of feature points from the captured image, wherein
each feature point of the plurality of feature points indicates a portion of the image, and
the portion of the image indicates a feature of the captured image; and
a local motion vector detection unit configured to detect the plurality of local motion vectors based on a plurality of motions of the extracted plurality of feature points.
9. The endoscopic surgery system according to claim 8 , further comprising a global motion amount estimation unit configured to estimate a global motion amount indicating an overall motion of the captured image, wherein
the global motion amount is estimated based on the plurality of local motion vectors, and
the correction amount control unit is further configured to obtain the correction amount based on the global motion amount.
10. The endoscopic surgery system according to claim 1 , wherein
the motion estimation unit is further configured to sequentially transmit the degree of reliability to the correction amount control unit, and
the correction amount control unit is further configured to:
accumulate the sequentially transmitted degree of reliability;
smooth the accumulated degree of reliability in a time direction; and
control the correction amount based on the smoothed degree of reliability.
11. The endoscopic surgery system according to claim 1 , wherein the correction processing unit is further configured to:
execute a correction process on the captured image based on the controlled correction amount; and
correct the blur based on the execution of the correction process.
12. The endoscopic surgery system according to claim 1 , wherein the surgical tool corresponds to at least one of an energy treatment tool or a forceps.
13. An image processing apparatus, comprising:
a motion estimation unit configured to:
acquire an image from an endoscope, wherein the endoscope captures the image;
estimate one or more motion components at a time of the capture of the image, wherein the one or more motion components are estimated based on the acquired; and
set, based on the estimated one or more motion components, a degree of reliability indicating that the estimated one or more motion components correspond to at least one of a motion of the endoscope or a motion of a surgical tool;
a correction amount control unit configured to control a correction amount based on the degree of reliability; and
a correction processing unit configured to correct, based on the controlled correction amount, a blur associated with the acquired image.
14. An image processing method, comprising:
in an image processing apparatus:
acquiring an image from an endoscope, wherein the endoscope captures the image;
estimating one or more motion components at a time of the capture of the image, wherein the one or more motion components are estimated based on the acquired image;
setting, based on the estimated one or more motion components, a degree of reliability indicating that the estimated one or more motion components correspond to at least one of a motion of the endoscope or a motion of a surgical tool;
controlling a correction amount based on the degree of reliability; and
correcting, based on the controlled correction amount, a blur associated with the acquired image.
15. An endoscopic surgery system, comprising:
an endoscope configured to capture an image;
a motion estimation unit configured to:
estimate one or more motion components at a time of the capture of the image, wherein the one or more motion components are estimated based on the captured image;
set, based on the estimated one or more motion components, a degree of reliability indicating that the estimated one or more motion components correspond to at least one of a motion of the endoscope or a motion of a surgical tool; and
sequentially transmit the degree of reliability;
a correction amount control unit configured to:
accumulate the sequentially transmitted degree of reliability;
smooth the accumulated degree of reliability in a time direction; and
control a correction amount based on the smoothed degree of reliability; and
a correction processing unit configured to correct, based on the controlled correction amount, a blur associated with the captured image.