Navigation of tubular networks
Methods and apparatuses provide improved navigation through tubular networks such as lung airways by providing improved estimation of location and orientation information of a medical instrument (e.g., an endoscope) within the tubular network. Various input data such as image data, EM data, and robot data are used by different algorithms to estimate the state of the medical instrument, and the state information is used to locate a specific site within a tubular network and/or to determine navigation information for what positions/orientations the medical instrument should travel through to arrive at the specific site. Probability distributions together with confidence values are generated corresponding to different algorithms are used to determine the medical instrument's estimated state.
1. A surgical robotic system comprising:
an endoscopic tool having a flexible tip;
an optical sensor coupled to the flexible tip;
a processor coupled to the optical sensor, the processor configured with instructions to:
receive a plurality of images from the optical sensor;
identify, within an image of the plurality of images, a division of a tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;
track the detected plurality of openings over a set of sequential image frames of the plurality of images to determine probabilities for entering each of the plurality of openings;
determine a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network and the probabilities, wherein the first estimated state comprises at least one of a first estimated position and a first estimated orientation of the flexible tip within the tubular network; and
show, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the tubular network, wherein the estimated position is determined based on the first estimated state.
2. The system of claim 1 , wherein the processor is configured to identify the division with the image by detecting the plurality of openings within the image.
3. The system of claim 2 , wherein processor is configured to detect the plurality of openings based on light reflective intensity within the image.
4. The system of claim 2 , wherein processor is configured to:
access model data from a three dimensional model of the tubular network; and
compare the detected plurality of openings with virtual branch information from the model data.
5. The system of claim 4 , wherein the three dimensional model of the tubular network is generated based on computerized axial tomography (CT) scans of the tubular network.
6. The system of claim 4 , wherein the processor is configured to compare the detected plurality of openings with the virtual branch information from the model data by comparing the detected plurality of openings with a virtual image of the division, wherein the virtual image is generated based on the model data.
7. The system of claim 6 , wherein the processor is configured to:
show the image on the display;
rotate the virtual image to match an orientation of the image; and
show the virtual image on the display.
8. The system of claim 2 , wherein the processor is configured to access model data comprising a tree diagram representative of the tubular network and determine the first estimated state based on the tree diagram.
9. The system of claim 1 , wherein the plurality of openings comprise three openings and the division comprises a trifurcation.
10. The system of claim 1 , wherein the plurality of images are captured sequentially in time.
11. The system of claim 10 , wherein the processor is configured to track relative positions of the detected plurality of openings over a set of sequential image frames of the plurality of images to determine a measurement of roll of the flexible tip.
12. The system of claim 1 , wherein the processor is configured to:
determine a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and
determine the estimated position of the flexible tip based on the first estimated state and the second estimated state.
13. The system of claim 12 , wherein the second set of sensor data comprises data derived from a position sensor.
14. The system of claim 12 , wherein the second set of sensor data comprises data regarding physical manipulation of the endoscopic tool.
15. A non-transitory computer readable storage medium comprising computer program instructions that when executed by one or more processors cause the one or more processors to:
receive a plurality of images from an optical sensor on a flexible tip of an endoscopic tool inserted into a tubular network;
identify, within an image of the plurality of images, a division of the tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;
track the detected plurality of openings over a set of sequential image frames of the plurality of images to determine probabilities for entering each of the plurality of openings;
determine a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network and the probabilities, wherein the first estimated state comprises at least one of a first estimated position and a first estimated orientation of the flexible tip within the tubular network; and
show, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the tubular network, wherein the estimated position is determined based on the first estimated state.
16. The non-transitory computer readable storage medium of claim 15 , wherein the instructions, when executed, cause the one or more processors to:
determine a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and
determine the estimated position of the flexible tip based on the first estimated state and the second estimated state.
17. The non-transitory computer readable storage medium of claim 16 , wherein the second set of sensor data comprises data derived from a position sensor.
18. A method for navigating a tubular network, the method comprising:
receiving a plurality of images from an optical sensor on a flexible tip of an endoscopic tool inserted into a tubular network;
identifying, within an image of the plurality of images, a division of the tubular network, the division comprising a plurality of openings to a plurality of respective segments of the tubular network;
tracking the detected plurality of openings over a set of sequential image frames of the plurality of images to determine probabilities for entering each of the plurality of openings;
determining a first estimated state of the flexible tip within the tubular network based on the identified division of the tubular network and the probabilities, wherein the first estimated state comprises at least one of a first estimated position and a first estimated orientation of the flexible tip within the tubular network; and
showing, on a display, an estimated position of the flexible tip within a segment of the plurality of segments of the tubular network, wherein the estimated position is determined based on the first estimated state.
19. The method of claim 18 , further comprising:
determining a second estimate state of the position of the device within the tubular network based on a second set of sensor data; and
determining the estimated position of the flexible tip based on the first estimated state and the second estimated state.