Endoscopic image processing apparatus, endoscope system, and method of operating endoscopic image processing apparatus

1. An endoscopic image processing apparatus comprising:

one or more processors configured to:

acquire an endoscopic image group obtained by causing an image pickup device provided at a distal end portion of an elongated insertion portion to pick up images of an inside of an object;

estimate a self-position of the image pickup device based on the endoscopic image group;

calculate a first displacement amount corresponding to a displacement amount of the image pickup device based on an estimation result of the self-position of the image pickup device obtained by the estimation;

acquire, from an insertion/removal state detection device that detects an insertion/removal state with respect to the object of the insertion portion inserted into the object, a detection signal indicating the insertion/removal state;

calculate a second displacement amount corresponding to a displacement amount in a direction parallel to a longitudinal axis direction of the insertion portion, based on the detection signal; and

generate scale information in which the first displacement amount and the second displacement amount are associated with each other.

2. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to:

detect a displacement direction of the insertion portion based on the detection signal;

in a case where the detected displacement direction of the insertion portion is a removal direction, perform processing to generate the scale information; and

in a case where the detected displacement direction of the insertion portion is an insertion direction, not perform the processing to generate the scale information.

3. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to:

calculate a first displacement speed corresponding to a displacement speed of the image pickup device based on the first displacement amount;

calculate a second displacement speed corresponding to a displacement speed of the insertion portion based on the second displacement amount; and

determine whether to perform processing to generate the scale information, based on correlation between variation of the first displacement speed in a predetermined time width and variation of the second displacement speed in the predetermined time width.

4. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors is configured to, in a case where control to direct the distal end portion to a direction intersecting the longitudinal axis direction of the insertion portion is performed, not perform processing to generate the scale information.

5. The endoscopic image processing apparatus according to claim 1 ,

wherein the insertion/removal state detection device is disposed at a position where a relative position to the object into which the insertion portion is inserted is not changed.

6. The endoscopic image processing apparatus according to claim 5 ,

wherein the insertion/removal state detection device is fixed at a predetermined position near an insertion port for insertion of the insertion portion into the object.

7. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to generate the scale information in which the first displacement amount having unknown scale and the second displacement amount are associated with each other, as the information used for the processing relating to creation of a three-dimensional shape model of the object.

8. An endoscope system, comprising:

an endoscope comprising an insertion portion and an image pickup device provided at a distal end portion of the insertion portion;

an insertion/removal state detection device configured to detect an insertion/removal state of the insertion portion inserted into an object and to output a detection signal; and

one or more processors configured to:

acquire an endoscopic image group obtained by causing the image pickup device to pick up images of an inside of the object;

estimate a self-position of the image pickup device based on the endoscopic image group;

calculate a first displacement amount corresponding to a displacement amount of the image pickup device based on an estimation result of the self-position of the image pickup device obtained by the estimation;

calculate a second displacement amount corresponding to a displacement amount in a direction parallel to a longitudinal axis direction of the insertion portion, based on the detection signal outputted from the insertion/removal state detection device; and

generate scale information in which the first displacement amount and the second displacement amount are associated with each other.

9. A method comprising:

acquiring an endoscopic image group obtained by causing an image pickup device provided at a distal end portion of an elongated insertion portion to pick up images of an inside of an object;

estimating a self-position of the image pickup device based on the endoscopic image group;

calculating a first displacement amount corresponding to a displacement amount of the image pickup device based on an estimation result of the self-position of the image pickup device obtained by the estimation;

acquiring, from an insertion/removal state detection device that detects an insertion/removal state with respect to the object of the insertion portion inserted into the object, a detection signal indicating the insertion/removal state;

calculating a second displacement amount corresponding to a displacement amount in a direction parallel to a longitudinal axis direction of the insertion portion, based on the detection signal; and

generating scale information in which the first displacement amount and the second displacement amount are associated with each other.

10. A non-transitory computer-readable storage medium storing instructions that cause a computer to at least perform:

acquiring an endoscopic image group obtained by causing an image pickup device provided at a distal end portion of an elongated insertion portion to pick up images of an inside of an object;

estimating a self-position of the image pickup device based on the endoscopic image group;

calculating a first displacement amount corresponding to a displacement amount of the image pickup device based on an estimation result of the self-position of the image pickup device obtained by the estimation;

acquiring, from an insertion/removal state detection device that detects an insertion/removal state with respect to the object of the insertion portion inserted into the object, a detection signal indicating the insertion/removal state;

calculating a second displacement amount corresponding to a displacement amount in a direction parallel to a longitudinal axis direction of the insertion portion, based on the detection signal; and

generating scale information in which the first displacement amount and the second displacement amount are associated with each other.

11. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to:

extract a plurality of matching feature points in the endoscopic image group; and

estimate the self-position of the image pickup device based on the plurality of matching feature points.

12. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to specify a three-dimensional coordinate position of the image pickup device in a world coordinate system of a three-dimensional space in which a three-dimensional shape model of the object is created, based on the estimation result of the self-position of the image pickup device.

13. The endoscopic image processing apparatus according to claim 12 ,

wherein the one or more processors are configured to specify the three-dimensional coordinate position of the image pickup device based on the estimation result of the self-position of the image pickup device and the scale information.

14. The endoscopic image processing apparatus according to claim 12 ,

wherein the one or more processors are configured to acquire three-dimensional point group coordinates including the three-dimensional coordinate position of the world coordinate system corresponding to the plurality of feature points based on the three-dimensional coordinate position.

15. The endoscopic image processing apparatus according to claim 12 ,

wherein the one or more processors are configured to create the three-dimensional shape model of the object based on at least one of the three-dimensional point group coordinates.

16. The endoscopic image processing apparatus according to claim 1 ,

wherein the first displacement amount is a displacement amount between a first estimation result of the self-position of the image pickup device acquired at a first timing and a second estimation result of the self-position of the image pickup device at a second timing, the first timing being different from the second timing.

17. The endoscopic image processing apparatus according to claim 1 ,

wherein the insertion/removal detection device includes a roller, and the detection signal corresponds to a rotation amount and a rotation direction of the roller.

18. The endoscopic image processing apparatus according to claim 1 ,

wherein the one or more processors are configured to generate a synchronization signal to synchronize a cycle in which the image pickup device outputs an image pickup signal with a cycle in which the insertion/removal detection device outputs the detection signal.