Biological image processing apparatus, method, non-transitory computer readable medium and biological image processing system
A surgical assistance apparatus includes circuitry configured to generate a first image to be displayed in a first display region and including a first operative field image of a surgical subject and a first visual annotation image superimposed on a predetermined region in an operative field of the first operative field image that corresponds to a physical region of the surgical subject. The circuitry is also configured to generate a second image to be displayed in a second display region and including a second operative field image and a second visual annotation image superimposed on an estimated region of the second operative field image that corresponds to the physical region of the surgical subject.
1 . A biological image processing apparatus comprising:
circuitry configured to
generate three-dimensional information indicating a three-dimensional structure of a biological subject;
receive, during a procedure, a first visual annotation image on a specified position in an operative field of a first operative field image that corresponds to a physical region of the biological subject;
according to a freeze mode input from a user interface device, generate a first image as a still image obtained from a moving image of an inside of the biological subject, wherein the first image includes the first operative field image of the biological subject and the first visual annotation image superimposed on the specified position;
according to a writing mode input from the user interface device, receive information associating the first visual annotation image with the physical region of the biological subject;
estimate a position of an imager that captures the second operative field image of the biological subject, wherein the imager is included in an endoscope that captures the second operative field image as a moving image of an inside of the biological subject; and
generate a second image by superimposing a second visual annotation image on a portion of the three-dimensional structure of the biological subject that corresponds to an estimated region of the second operative field image that is estimated to correspond to the physical region of the biological subject based on the estimated movement of the imager and the generated three-dimensional information, wherein the second operative field image is an image subsequent to the first operative field image.
2 . The biological image processing apparatus according to claim 1 , wherein:
the generated three-dimensional information of the biological subject includes time information indicating a generation/update time of the corresponding three-dimensional information.
3 . The biological image processing apparatus according to claim 1 , wherein:
the first visual annotation image is input by an input device configured to be used by a first user to input the first visual annotation image on a first display region that displays the first image without obscuring a view of a second display region that displays the second image by a second user.
4 . The biological image processing apparatus according to claim 3 , wherein:
the input device to input the first visual annotation image is configured to be used by the first user to identify the physical region of the biological subject as a target for an action to be performed on the biological subject; and
the second visual annotation image in the second display region is configured to be used by the second user to control execution of the action on the physical region of the biological subject.
5 . The biological image processing apparatus according to claim 3 , wherein:
the first display region is displayed on a first display device; and
the second display region is displayed on a second display device.
6 . The biological image processing apparatus according to claim 3 , wherein:
the first display region is displayed in a first window area of a display; and
the second display region is displayed in a second window area of the display.
7 . The biological image processing apparatus according to claim 1 , wherein the circuitry is further configured to update the second image to maintain the correspondence between the second visual annotation superimposed on the estimated region of the second operative field image and the physical region of the subject during a movement of the imager or a movement of the physical region of the biological subject.
8 . The biological image processing apparatus according to claim 1 , wherein the circuitry includes a programmable processor.
9 . The biological image processing apparatus according to claim 1 , wherein the second operative field image includes a moving image of the biological subject that indicates a movement of the biological subject in real-time.
10 . The biological image processing apparatus according to claim 9 , wherein the circuitry is further configured to obtain the first operative field image from the moving image of the biological subject.
11 . The biological image processing apparatus according to claim 1 , wherein the circuitry is further configured to receive information regarding the first visual annotation image from an input device operated using a first display region that display the first image.
12 . The biological image processing apparatus according to claim 1 , wherein the second operative field image is a moving image that indicates a movement of at least the biological subject and the second visual annotation image moves with the movement of at least the biological subject to maintain the correspondence between the estimated region of the second operative field image and the physical region of the biological subject.
13 . The biological image processing apparatus according to claim 1 , wherein the circuitry is further configured to:
generate the second visual annotation image including an indication of a position in a depth direction of the second operative field image on the basis of the generated three-dimensional information.
14 . The biological image processing apparatus according to claim 1 , wherein the circuitry is further configured to:
conduct a masking display on a region of the first image in which the generated three-dimensional information has not been sufficiently constructed.
15 . The biological image processing apparatus according to claim 14 , wherein the circuitry is further configured to:
in a case where the estimated region of the second operation field image that corresponds to the physical region of the biological subject includes the region of the first image in which the generated three-dimensional information has not been sufficiently constructed, the second image is generated without superimposing at least a portion of the second visual annotation image.
16 . The biological image processing apparatus according to claim 1 , wherein the biological subject is a surgical subject.
17 . The biological image processing apparatus according to claim 1 , wherein the first image is a still image and the second image is a live image.
18 . A biological image processing apparatus comprising:
circuitry configured to
receive, during a procedure, a first visual annotation image on a specified position in an operative field of a first operative field image that corresponds to a physical region of a biological subject;
generate a first image and including the first operative field image of the biological subject and the first visual annotation image superimposed on the specified position,
track the specified position over time;
generate a second image and including a second operative field image and a second visual annotation image superimposed on estimated region of the second operative field image that corresponds the physical region of the biological subject to the physical region of the biological subject based on a track of the specified position such that the second visual annotation image is superimposed in a same position on the physical region of the biological subject as the first visual annotation image, wherein the second operative field image is an image subsequent to the first operative field image;
conduct a masking display on a region of the first image in which the generated three-dimensional information has not been sufficiently constructed;
in a case where the estimated region of the second operation field image that corresponds to the physical region of the biological subject includes the region of the first image in which the generated three-dimensional information has not been sufficiently constructed, the second image is generated without superimposing at least a portion of the second visual annotation image; and
in a case where the estimated region of the second operation field image that corresponds to the physical region of the biological subject includes the region of the first image in which the generated three-dimensional information has not been sufficiently constructed, the first image is displayed as a still image instead of a moving image.
19 . A endoscopic image processing system comprising:
an endoscope including an imager that captures a second operative field image of a biological subject;
circuitry configured to
generate three-dimensional information indicating a three-dimensional structure of the biological subject:
receive, during a procedure, a first visual annotation image on a specified position in an operative field of a first operative field image that corresponds to a physical region of the biological subject;
according to a freeze mode input from a user interface device, generate a first image as a still image obtained from a moving image of an inside of the biological subject, wherein the first image includes the first operative field image of the biological subject and the first visual annotation image superimposed on the specified position;
according to a writing mode input from the user interface device, receive information associating the first visual annotation image with the physical region of the biological subject;
generate a second image by superimposing a second visual annotation image on a portion of the three-dimensional structure of the biological subject that corresponds to an estimated region of the second operative field image that is estimated to correspond to the physical region of the biological subject based on the estimated movement of the imager and the generated three-dimensional information; and
a surgical tool that is moved inside the biological subject to perform a surgical procedure on the biological subject under the control of a healthcare worker based on the second visual annotation image in the second image based on a track of the specified position, wherein the second operative field image is an image subsequent to the first operative field image.