Image-based branch detection and mapping for navigation
Navigation of an instrument within a luminal network can include image-based branch detection and mapping. Image-based branch detection can include identifying within an image one or more openings associated with one or more branches of a luminal network. Image-based branch mapping can include mapping the detected one or more openings to corresponding branches of the luminal network. Mapping may include comparing features of the openings to features of a set of expected openings. A position state estimate for the instrument can be determined from the mapped openings, which can facilitate navigation of the luminal network.
1. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause a processor of a device to:
receive a position state estimate of an instrument positioned within a current branch of a luminal network;
receive an image of an interior of the luminal network from an imaging device positioned on the instrument; and
detect within the image a first opening connecting a first subsequent branch of the luminal network to the current branch and a second opening connecting a second subsequent branch of the luminal network to the current branch, said detecting involving;
generating a histogram of the image,
identifying a first peak of the histogram as corresponding to pixels associated with airway openings;
identifying a second peak of the histogram as corresponding to pixels associated with biological tissue;
determining a threshold value between the first peak and the second peak; and
identifying pixels below the threshold value as representing the first and second openings.
2. The non-transitory computer readable storage medium of claim 1 , further having stored thereon instructions that, when executed, cause the processor to:
determine which of the first subsequent branch and the second subsequent branch is closest to a center of the image, and
determine a first probability that the instrument will enter the first subsequent branch and determine a second probability that the instrument will enter the second subsequent branch based on which of the first subsequent branch and the second subsequent branch is closest to the center of the image.
3. The non-transitory computer readable storage medium of claim 1 , further having stored thereon instructions that, when executed, cause the processor to:
identify a first centroid of the first subsequent branch in the image;
identify a second centroid of the second subsequent branch in the image;
determine a first distance between the first centroid and a center of the image;
determine a second distance between the second centroid and the center of the image,
determine a first probability that the instrument will enter the first subsequent branch based on the first distance, and
determine a second probability that the instrument will enter the second subsequent branch based on the second distance.
4. The non-transitory computer readable storage medium of claim 3 , wherein the first probability is proportionally related to the first distance and the second probability is proportionally related to the second distance.
5. The non-transitory computer readable storage medium of claim 1 , further having stored thereon instructions that, when executed, cause the processor to:
determine a first expected subsequent branch of the luminal network and a second expected subsequent branch of the luminal network based on the received position state estimate and a preoperative model of the luminal network;
determine a feature of the first opening and the second opening;
determine an expected feature of the first expected subsequent branch and the second expected subsequent branch;
compare the feature to the expected feature to map the first opening and the second opening to the first expected subsequent branch and the second expected subsequent branch; and
determine an updated position state estimate is further based on the mapping of the first opening and the second opening.
6. The non-transitory computer readable storage medium of claim 5 , wherein the updated position state estimate comprises a probability that the updated position state estimate is correct.
7. The non-transitory computer readable storage medium of claim 6 , wherein the probability is based on the comparing of the feature of the first opening and the second opening to the expected feature of the first expected subsequent branch and the second expected subsequent branch.
8. The non-transitory computer readable storage medium of claim 5 , wherein:
the updated position state estimate comprises an estimate of roll of the instrument about a longitudinal axis of the instrument,
the updated position state estimate comprises a probability that the estimate of roll is correct, and
the probability is based on comparing an orientation of the first opening and the second opening to an expected orientation of the first expected subsequent branch and the second expected subsequent branch.
9. A robotic system, comprising:
an instrument having an elongate body configured to be inserted into a luminal network of a patient, and an imaging device positioned on a distal portion of the elongate body;
an instrument positioning device attached to the instrument, the instrument positioning device configured to move the instrument through the luminal network;
at least one computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to:
receive a position state estimate of the instrument positioned within a current branch of the luminal network,
receive an image of an interior of the luminal network from the imaging device, and
detect within the image a first opening connecting a first subsequent branch of the luminal network to the current branch and a second opening connecting a second subsequent branch of the luminal network to the current branch, said detecting involving:
generating a histogram of the image,
identifying a first peak of the histogram as corresponding to pixels associated with airway openings,
identifying a second peak of the histogram as corresponding to pixels associated with biological tissue,
determining a threshold value between the first peak and the second peak, and
identifying pixels below the threshold value as representing the first and second openings.
10. The system of claim 9 , wherein the at least one computer-readable memory further has stored thereon executable instructions that, when executed by the one or more processors, cause the system to:
identify a first centroid of the first opening in the image,
identify a second centroid of the second opening in the image, and
determine which of the first opening and the second opening is closest to a center of the image by:
determining a first distance between the first centroid and the center of the image, and
determining a second distance between the second centroid and the center of the image.
11. The system of claim 10 , wherein the at least one computer-readable memory further has stored thereon executable instructions that, when executed by the one or more processors, cause the system to determine a first probability that the instrument will enter the first subsequent branch, the first probability being proportionally related to the first distance.
12. The system of claim 9 , wherein the at least one computer-readable memory further has stored thereon executable instructions that, when executed by the one or more processors, cause the system to:
determine a first expected subsequent branch of the luminal network and a second expected subsequent branch of the luminal network based on the received position state estimate and a preoperative model of the luminal network;
determine a feature of the first opening and the second opening;
determine an expected feature of the first expected subsequent branch and the second expected subsequent branch;
compare the feature to the expected feature to map the first opening and the second opening to the first expected subsequent branch and the second expected subsequent branch, and
determine an updated position state estimate based on the mapping of the first opening and the second opening.
13. The system of claim 12 , wherein the updated position state estimate comprises a probability that the updated position state estimate is correct.
14. The system of claim 13 , wherein the probability is based on the comparing of the feature of the first opening and the second opening to the expected feature of the first expected subsequent branch and the second expected subsequent branch.
15. The system of claim 12 , wherein:
the updated position state estimate comprises an estimate of roll of the instrument about a longitudinal axis of the instrument,
the updated position state estimate comprises a probability that the estimate of roll is correct, and
the probability is based on comparing an orientation of the first opening and the second opening to an expected orientation of the first expected subsequent branch and the second expected subsequent branch.
16. A method for navigating an instrument through a luminal network, the method comprising:
receiving a position state estimate of the instrument positioned within a current branch of the luminal network;
receiving an image of an interior of the luminal network from an imaging device positioned on the instrument; and
detecting within the image a first opening connecting a first subsequent branch of the luminal network to the current branch and a second opening connecting a second subsequent branch of the luminal network to the current branch, said detecting involving:
generating a histogram of the image,
identifying a first peak in the histogram as corresponding to pixels associated with airway openings;
identifying a second peak in the histogram as corresponding to pixels associated with biological tissue;
determining a threshold value between the first peak and the second peak; and
identifying pixels below the threshold value as representing the first and second openings.
17. The method of claim 16 , further comprising:
determining which of the first subsequent branch and the second subsequent branch is closest to a center of the image, and
determining a first probability that the instrument will enter the first subsequent branch and a second probability that the instrument will enter the second subsequent branch based on which of the first subsequent branch and the second subsequent branch is closest to the center of the image.
18. The method of claim 16 , further comprising:
identifying a first centroid of the first subsequent branch in the image;
identifying a second centroid of the second subsequent branch in the image;
determining a first distance between the first centroid and a center of the image;
determining a second distance between the second centroid and the center of the image,
determining a first probability that the instrument will enter the first subsequent branch based on the first distance, and
determining a second probability that the instrument will enter the second subsequent branch based on the second distance.
19. The method of claim 18 , wherein the first probability is proportionally related to the first distance and the second probability is proportionally related to the second distance.