Automatically-initialized robotic systems for navigation of luminal networks
Certain aspects relate to systems and techniques for navigation-assisted medical devices. Some aspects relate to correlating features of depth information generated based on captured images of an anatomical luminal network with virtual features of depth information generated based on virtual images of a virtual representation of the anatomical luminal network in order to automatically determine an initial position of an endoscope within the luminal network.
1. A method of facilitating navigation of an anatomical luminal network of a patient, the method, executed by a set of one or more computing devices, comprising:
receiving imaging data captured by an imaging device at a distal end of an instrument positioned within the anatomical luminal network;
accessing a virtual feature derived from a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network, wherein the virtual feature represents a first spatial relationship between two or more depth criteria in the virtual depth map;
generating a depth map based on the imaging data;
deriving a feature from the depth map, the feature representing a second spatial relationship between two or more depth criteria in the depth map;
calculating a correspondence between the feature derived from the depth map and the virtual feature derived from the virtual depth map; and
determining a pose of the distal end of the instrument within the anatomical luminal network based on the virtual location associated with the virtual feature based on the calculated correspondence.
2. The method of claim 1 , further comprising:
generating the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value representing an estimated distance between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixel;
identifying a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map;
calculating the feature as a first value representing a distance between the first and second pixels;
wherein the virtual depth map comprises, for each virtual pixel of a plurality of virtual pixels, a virtual depth value representing a virtual distance between the virtual imaging device and a portion of the virtual luminal network represented by the virtual pixel, and wherein accessing the virtual feature derived from the virtual image comprises accessing a second value representing a distance between first and second depth criteria in the virtual depth map; and
calculating the correspondence based on comparing the first value to the second value.
3. The method of claim 2 , further comprising:
accessing a plurality of values representing distances between first and second depth criteria in a plurality of virtual depth maps each representing a different one of a plurality of virtual locations within the virtual luminal network; and
calculating the correspondence based on the second value corresponding more closely to the first value than other values of the plurality of values.
4. The method of claim 2 , wherein the anatomical luminal network comprises airways and the imaging data depicts a bifurcation of the airways, the method further comprising:
identifying one of the first and second depth criteria as a right bronchus in each of the depth map and the virtual depth map; and
determining a roll of the instrument based on an angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein the pose of the distal end of the instrument within the anatomical luminal network comprises the determined roll.
5. The method of claim 1 , further comprising:
identifying three or more depth criteria in each of the depth map and the virtual depth map;
determining a shape and location of a polygon connecting the depth criteria in each of the depth map and the virtual depth map; and
calculating the correspondence based on comparing the shape and location of the polygon of the depth map to the shape and location of the polygon of the virtual depth map.
6. The method of claim 1 , wherein generating the depth map is based on photoclinometry.
7. The method of claim 1 , further comprising:
calculating a probabilistic state of the instrument within the anatomical luminal network based on a plurality of inputs comprising the pose of the distal end of the instrument within the anatomical luminal network; and
guiding navigation of the instrument through the anatomical luminal network based at least partly on the probabilistic state.
8. The method of claim 7 , further comprising initializing a navigation system configured to calculate the probabilistic state and guide the navigation of the anatomical luminal network based on the probabilistic state, wherein the initializing of the navigation system comprises setting a prior of a probability calculator based on the pose of the distal end of the instrument within the anatomical luminal network.
9. The method of claim 8 , further comprising:
receiving additional data representing an updated pose of the distal end of the instrument;
setting a likelihood function of the probability calculator based on the additional data; and
determining the probabilistic state using the probability calculator based on the prior and the likelihood function.
10. The method of claim 1 , further comprising determining a registration between a coordinate frame of the virtual luminal network and a coordinate frame of an electromagnetic field generated around the anatomical luminal network based at least partly on the pose of the distal end of the instrument within the anatomical luminal network determined based on the calculated correspondence.
11. A system configured to facilitate navigation of an anatomical luminal network of a patient, the system comprising:
an imaging device at a distal end of an instrument;
at least one computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least:
receive imaging data captured by the imaging device with the distal end of the instrument positioned within the anatomical luminal network;
access a virtual feature derived from a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network, wherein the virtual feature represents a first spatial relationship between two or more depth criteria in the virtual depth map;
generate a depth map based on the imaging data;
derive a feature from the depth map, the feature representing a second spatial relationship between two or more depth criteria in the depth map;
calculate a correspondence between the feature derived from the depth map and the virtual feature derived from the virtual depth; and
determine a pose of the distal end of the instrument relative within the anatomical luminal network based on the virtual location associated with the virtual feature based on the calculated correspondence.
12. The system of claim 11 , wherein the virtual image represents the virtual depth map.
13. The system of claim 11 , wherein the one or more processors are configured to execute the instructions to cause the system to at least:
generate the depth map by calculating, for each pixel of a plurality of pixels of the imaging data, a depth value representing an estimated distance between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixel;
identify a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map and a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map;
calculate the feature as a first value representing a distance between the first and second pixels;
wherein the virtual depth map comprises, for each virtual pixel of a plurality of virtual pixels, a virtual depth value representing a virtual distance between the virtual imaging device and a portion of the virtual luminal network represented by the virtual pixel, and wherein the virtual feature derived from the virtual image comprises a second value representing a distance between first and second depth criteria in the virtual depth map; and
determine the correspondence based on comparing the first value to the second value.
14. The system of claim 13 , wherein the one or more processors are configured to execute the instructions to cause the system to at least:
access a plurality of values representing distances between first and second depth criteria in a plurality of virtual depth maps each representing a different one of a plurality of virtual locations within the virtual luminal network; and
calculate the correspondence based on the second value corresponding more closely to the first value than other values of the plurality of values identify the second value as a closest match to the first value among the plurality of values.
15. The system of claim 13 , wherein the anatomical luminal network comprises airways and the imaging data depicts a bifurcation of the airways, wherein the one or more processors are configured to execute the instructions to cause the system to at least:
identify one of the first and second depth criteria as a right bronchus in each of the depth map and the virtual depth map; and
determine a roll of the instrument based on an angular distance between a first position of the right bronchus in the depth map and a second position of the right bronchus in the virtual depth map, wherein the pose of the distal end of the instrument within the anatomical luminal network comprises the determined roll.
16. The system of claim 11 , wherein the one or more processors are configured to execute the instructions to cause the system to at least:
identify three or more depth criteria in each of the depth map and the virtual depth map;
determine a shape and location of a polygon connecting the three or more depth criteria in each of the depth map and the virtual depth map; and
calculate the correspondence based on comparing the shape and location of the polygon of the depth map to the shape and location of the polygon of the virtual depth map.
17. The system of claim 11 , wherein the one or more processors are configured to communicate with a navigation system, and wherein the one or more processors are configured to execute the instructions to cause the system to at least:
calculate a probabilistic state of the instrument within the anatomical luminal network using the navigation system based at least partly on a plurality of inputs comprising the pose of the distal end of the instrument within the anatomical luminal network; and
guide navigation of the instrument through the anatomical luminal network based at least partly on the probabilistic state calculated by the navigation system.
18. The system of claim 17 , further comprising a robotic system configured to guide movements of the instrument during the navigation, wherein the plurality of inputs comprise robotic position data received from the robotic system, and wherein the one or more processors are configured to execute the instructions to cause the system to at least calculate the probabilistic state of the instrument using the navigation system based at least partly on the pose of the distal end of the instrument within the anatomical luminal network and on the robotic position data.
19. The system of claim 16 , further comprising a position sensor at the distal end of an instrument, the plurality of inputs comprise data received from the position sensor, and wherein the one or more processors are configured to execute the instructions to cause the system to at least calculate the probabilistic state of the instrument using the navigation system based at least partly on the pose of the distal end of the instrument within the anatomical luminal network and on the data received from the position sensor.
20. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to at least:
access a virtual three-dimensional model of internal surfaces of an anatomical luminal network of a patient;
identify a plurality of virtual locations within the virtual three-dimensional model;
for each virtual location of the plurality of virtual locations within the virtual three-dimensional model:
generate a virtual depth map representing virtual distances between a virtual imaging device positioned at the virtual location and a portion of the internal surfaces within a field of view of the virtual imaging device when positioned at the virtual location, and
derive at least one virtual feature from the virtual depth map, wherein the virtual feature represents a spatial relationship between two or more depth criteria in the virtual depth map; and
generate a database associating the plurality of virtual locations with the at least one virtual feature derived from the corresponding virtual depth map.
21. The non-transitory computer readable storage medium of claim 20 , wherein the instructions, when executed, cause the at least one computing device to at least provide the database to a navigation system configured to guide navigation of an instrument through the anatomical luminal network during a medical procedure.
22. The non-transitory computer readable storage medium of claim 21 , wherein the instructions, when executed, cause the at least one computing device to at least:
access data representing an imaging device positioned at a distal end of the instrument;
identify image capture parameters of the imaging device; and
set virtual image capture parameters of the virtual imaging device to correspond to the image capture parameters of the imaging device.
23. The non-transitory computer readable storage medium of claim 22 , wherein the instructions, when executed, cause the at least one computing device to at least generate the virtual depth maps based on the virtual image capture parameters.
24. The non-transitory computer readable storage medium of claim 22 , wherein the image capture parameters comprise one or more of field of view, lens distortion, focal length, and brightness shading.
25. The non-transitory computer readable storage medium of claim 20 , wherein the instructions, when executed, cause the at least one computing device to at least:
for each virtual location of the plurality of virtual locations:
identify first and second depth criteria in the virtual depth map, and
calculate the virtual feature as a value representing a distance between the first and second depth criteria; and
create the database by associating the plurality of virtual locations with the corresponding value.
26. The non-transitory computer readable storage medium of claim 20 , wherein the instructions, when executed, cause the at least one computing device to at least:
for each virtual location of the plurality of virtual locations:
identify three or more depth criteria in the virtual depth map, and
determine the virtual feature as a shape and location of a polygon connecting the three or more depth criteria; and
create the database by associating the plurality of virtual locations with the shape and location of the corresponding polygon.
27. A method of facilitating navigation of an anatomical luminal network of a patient, the method, executed by a set of one or more computing devices, comprising:
receiving a stereoscopic image set representing an interior of the anatomical luminal network;
generating a depth map based on the stereoscopic image set;
deriving a feature from the depth map, the feature representing a second spatial relationship between two or more depth criteria in the depth map;
accessing a virtual feature derived from a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network, wherein the virtual feature represents a first spatial relationship between two or more depth criteria in the virtual depth map;
calculating a correspondence between the feature derived from the depth map and the virtual feature derived from the virtual depth map; and
determining a pose of the distal end of the instrument within the anatomical luminal network based on the virtual location associated with the virtual feature based on the calculated correspondence.
28. The method of claim 27 , wherein generating the stereoscopic image set comprises:
positioning an imaging device at a distal end of an instrument at a first location within the anatomical luminal network;
capturing a first image of an interior of the anatomical luminal network with the imaging device positioned at the first location;
robotically controlling the imaging device to move a known distance to a second location within the anatomical luminal network; and
capturing a second image of the interior of the anatomical luminal network with the imaging device positioned at the second location.
29. The method of claim 28 , wherein robotically controlling the imaging device to move the known distance comprises one or both of retracting the imaging device and angularly rolling the imaging device.
30. The method of claim 29 , wherein the virtual feature comprises a value representing a first distance between first and second depth criteria of the virtual depth map, the method further comprising calculating the feature derived from the depth map as a second distance between first and second depth criteria of the depth map.