Robotic systems for determining a roll of a medical device in luminal networks
Certain aspects relate to systems and techniques for navigation-assisted medical devices. Some aspects relate to correlating features of depth information generated based on captured images of an anatomical luminal network with virtual features of depth information generated based on virtual images of a virtual representation of the anatomical luminal network in order to automatically determine aspects of a roll of a medical device within the luminal network.
1. A method of facilitating navigation of an anatomical luminal network of a patient, the method, executed by a set of one or more computing devices, comprising:
receiving imaging data captured by an imaging device at a distal end of an instrument positioned within the anatomical luminal network;
generating a depth map based on the imaging data;
accessing a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network;
identifying three or more depth criteria in each of the depth map and the virtual depth map;
determining a first shape connecting the three or more depth criteria in the depth map and a second shape connecting the three or more depth criteria in the virtual depth map;
determining that the first shape and the second shape match within a threshold difference; and
determining a pose of the distal end of the instrument within the anatomical luminal network based on the determination that the first shape and the second shape match within the threshold difference.
2. The method of claim 1 , wherein:
generating the depth map comprises determining, for a plurality of pixels of the imaging data, depth values representing estimated distances between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixels,
accessing the virtual depth map comprises accessing, for a plurality of virtual pixels of the virtual image, virtual depth values representing virtual distances between the virtual imaging device and a portion of the virtual luminal network represented by the virtual pixels,
identifying the three or more depth criteria in each of the depth map and the virtual depth map comprises:
identifying at least a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map, a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map, and a third pixel of the plurality of pixels corresponding to a third depth criterion, and
accessing at least a first virtual pixel of the plurality of virtual pixels corresponding to a first virtual depth criterion in the depth map, a second virtual pixel of the plurality of virtual pixels corresponding to a second virtual depth criterion in the depth map, and a third virtual pixel of the plurality of virtual pixels corresponding to a third virtual depth criterion; and
determining the first shape and the second shape comprises:
determining the first shape as a first polygon connecting at least the first pixel, the second pixel, and the third pixel, and
determining the second shape comprises determining the second shape as a second polygon connecting at least the first virtual pixel, the second virtual pixel, and the third virtual pixel.
3. The method of claim 2 , wherein:
the three or more depth criteria associated with the depth map corresponds to positions of local depth value extrema within the depth map, and
the three or more depth criteria associated with the virtual depth map corresponds to positions of local depth value extrema within the virtual depth map.
4. The method of claim 2 , wherein determining that the first shape and the second shape match within the threshold difference comprises comparing at least one of:
a first size of the first polygon and a second size of the second polygon,
a first location of the first polygon and a second location of the second polygon, or
a first orientation of the first polygon and a second orientation of the second polygon.
5. The method of claim 2 , further comprising:
generating a database associating a virtual location and the second shape derived from the corresponding virtual depth map, the virtual location within a three-dimensional model of internal surfaces of the anatomical luminal network.
6. The method of claim 5 , wherein the database further associates the virtual location with at least one of an airway segment or a depth within the airway segment.
7. The method of claim 1 , wherein generating the depth map is based on photoclinometry performed on a single captured image.
8. The method of claim 1 , wherein generating the depth map comprises:
positioning the imaging device at a first location within the anatomical luminal network;
capturing a first image of a set of stereoscopic images with the imaging device positioned at the first location;
robotically controlling the imaging device to move a known distance to a second location within the anatomical luminal network; and
capturing a second image of the set of stereoscopic images with the imaging device positioned at the second location.
9. The method of claim 8 , wherein robotically controlling the imaging device to move the known distance comprises one or both of retracting the imaging device and angularly rolling the imaging device.
10. The method of claim 1 , further comprising determining a registration between a coordinate frame of the virtual luminal network and a coordinate frame of an electromagnetic field generated around the anatomical luminal network based at least in part on the pose of the distal end of the instrument within the anatomical luminal network.
11. The method of claim 1 , wherein determining that the first shape and the second shape match within a threshold difference comprises identifying an angular transformation between the first shape and the second shape, and
the method further comprising:
determining a roll associated with the distal end of the instrument within the anatomical luminal network based on the angular transformation.
12. The method of claim 1 , wherein determining the first shape and the second shape comprises:
determining the first shape comprises determining the first shape as a first curve peak surrounding a first local depth value extrema within the depth map, and
determining the second shape comprises determining the second shape as a second curve peak surrounding a second local depth value extrema within the virtual depth map.
13. A system configured to facilitate navigation of an anatomical luminal network of a patient, the system comprising:
an imaging device at a distal end of an instrument positioned within the anatomical luminal network;
at least one computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one computer-readable memory and configured to execute the instructions to cause the system to at least:
receive imaging data captured by the imaging device at the distal end of the instrument;
generate a depth map based on the imaging data;
access a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of the anatomical luminal network;
identify three or more depth criteria in each of the depth map and the virtual depth map;
determine a first shape connecting the three or more depth criteria in the depth map and a second shape connecting the three or more depth criteria in the virtual depth map;
determine that the first shape and the second shape match within a threshold difference; and
determine a pose of the distal end of the instrument within the anatomical luminal network based on the determination that the first shape and the second shape match within the threshold difference.
14. The system of claim 13 , wherein:
generating the depth map comprises determining, for a plurality of pixels of the imaging data, depth values representing estimated distances between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixels,
accessing the virtual depth map comprises accessing, for a plurality of virtual pixels of the virtual image, virtual depth values representing virtual distances between the virtual imaging device and a portion of the virtual luminal network represented by the virtual pixels,
identifying the three or more depth criteria in each of the depth map and the virtual depth map comprises:
identifying at least a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map, a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map, and a third pixel of the plurality of pixels corresponding to a third depth criterion, and
accessing at least a first virtual pixel of the plurality of virtual pixels corresponding to a first virtual depth criterion in the depth map, a second virtual pixel of the plurality of virtual pixels corresponding to a second virtual depth criterion in the depth map, and a third virtual pixel of the plurality of virtual pixels corresponding to a third virtual depth criterion; and
determining the first shape and the second shape comprises:
determining the first shape as a first polygon connecting at least the first pixel, the second pixel, and the third pixel, and
determining the second shape comprises determining the second shape as a second polygon connecting at least the first virtual pixel, the second virtual pixel, and the third virtual pixel.
15. The system of claim 14 , wherein determining that the first shape and the second shape match within the threshold difference comprises comparing at least one of:
a first size of the first polygon and a second size of the second polygon,
a first location of the first polygon and a second location of the second polygon, or
a first orientation of the first polygon and a second orientation of the second polygon.
16. The system of claim 13 , wherein:
the three or more depth criteria associated with the depth map corresponds to positions of local depth value extrema within the depth map, and
the three or more depth criteria associated with the virtual depth map corresponds to positions of local depth value extrema within the virtual depth map.
17. The system of claim 13 , wherein generating the depth map is based on photoclinometry performed on a single captured image.
18. The system of claim 13 , wherein generating the depth map comprises:
positioning the imaging device at a first location within the anatomical luminal network;
capturing a first image of a set of stereoscopic images with the imaging device positioned at the first location;
robotically controlling the imaging device to move a known distance to a second location within the anatomical luminal network; and
capturing a second image of the set of stereoscopic images with the imaging device positioned at the second location.
19. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, cause at least one computing device to:
receive imaging data captured by an imaging device at a distal end of an instrument;
generate a depth map based on the imaging data;
access a virtual depth map associated with a virtual image, the virtual image simulated from a viewpoint of a virtual imaging device positioned at a virtual location within a virtual luminal network representative of an anatomical luminal network;
identify three or more depth criteria in each of the depth map and the virtual depth map;
determine a first shape connecting the three or more depth criteria in the depth map and a second shape connecting the three or more depth criteria in the virtual depth map;
determine that the first shape and the second shape match within a threshold difference; and
determine a pose of the distal end of the instrument within the anatomical luminal network based on the determination that the first shape and the second shape match within the threshold difference.
20. The non-transitory computer readable storage medium of claim 19 , wherein:
generating the depth map comprises determining, for a plurality of pixels of the imaging data, depth values representing estimated distances between the imaging device and a tissue surface within the anatomical luminal network corresponding to the pixels,
accessing the virtual depth map comprises accessing, for a plurality of virtual pixels of the virtual image, virtual depth values representing virtual distances between the virtual imaging device and a portion of the virtual luminal network represented by the virtual pixels,
identifying the three or more depth criteria in each of the depth map and the virtual depth map comprises:
identifying at least a first pixel of the plurality of pixels corresponding to a first depth criterion in the depth map, a second pixel of the plurality of pixels corresponding to a second depth criterion in the depth map, and a third pixel of the plurality of pixels corresponding to a third depth criterion, and
accessing at least a first virtual pixel of the plurality of virtual pixels corresponding to a first virtual depth criterion in the depth map, a second virtual pixel of the plurality of virtual pixels corresponding to a second virtual depth criterion in the depth map, and a third virtual pixel of the plurality of virtual pixels corresponding to a third virtual depth criterion; and
determining the first shape and the second shape comprises:
determining the first shape as a first polygon connecting at least the first pixel, the second pixel, and the third pixel, and
determining the second shape comprises determining the second shape as a second polygon connecting at least the first virtual pixel, the second virtual pixel, and the third virtual pixel.