Vision-based 6DOF camera pose estimation in bronchoscopy
Methods and systems provide improved navigation through tubular networks such as lung airways by providing improved estimation of location and orientation information of a medical instrument (e.g., an endoscope) within the tubular network. Various input data such as image data and CT data, are used to model the tubular networks, and the model information is used to generate a camera pose representing a specific site location within the tubular network and/or to determine navigation information including position and orientation for the medical instrument.
1. A method, comprising:
generating a camera pose representative of a spatial location within an internal anatomical structure and capturing video image data of the internal anatomical structure, comprising:
transforming the video image data into a first depth map of the internal anatomical structure, via a mapping function;
receiving a plurality of second depth maps based at least in part on transformed computed tomography (CT) image data, each of the second depth maps representing a virtual model of the internal anatomical structure;
identifying one of the plurality of second depth maps that has a highest similarity of shape to the first depth map; and
generating the camera pose based at least in part on the identified one of the plurality of second depth maps, including estimating camera pose parameters by solving a transformation matrix between a 3D point cloud of the virtual model of the internal anatomical structure and a point cloud inverse projected from the first depth map.
2. The method of claim 1 , further comprising forming the first depth map from the video image data using a convolutional neural network (CNN).
3. The method of claim 1 , wherein a convolutional neural network (CNN) is employed to solve the transformation matrix and estimate the camera pose parameters.
4. The method of claim 1 , further comprising identifying a desired location in the one of the plurality of second depth maps that corresponds to a location in one or more video images, using the first depth map.
5. The method of claim 1 , wherein the identifying one of the plurality of second depth maps is an iterative or continuous process.
6. The method of claim 5 , wherein the iterative process includes using the camera pose of one of the plurality of second depth maps of a previous iteration to initialize generating the plurality of second depth maps.
7. The method of claim 1 , further comprising generating the plurality of second depth maps using a plurality of virtual camera poses.
8. The method of claim 1 , wherein the camera pose comprises a virtual spatial estimation representative of a physical spatial location within the internal anatomical structure.
9. The method of claim 8 , further comprising defining the camera pose in coordinates relative to the CT image data.
10. A method, comprising:
generating a camera pose representative of a spatial location within a bronchial airway and capturing video image data of the bronchial airway, comprising:
transforming the video image data into a first depth map of the bronchial airway, via a mapping function;
receiving a plurality of second depth maps based at least in part on computed tomography (CT) image data, each of the second depth maps representing a virtual model of the internal anatomical shape structure of the bronchial airway;
identifying one of the plurality of second depth maps that has a highest similarity of shape to the first depth map;
estimating camera pose parameters by solving a transformation matrix between a 3D point cloud of the virtual model of the internal anatomical structure and a point cloud inverse projected from the first depth map; and
generating the camera pose representing a location and an orientation having six-degrees of freedom (6DoF) within the bronchial airway, based at least in part on the identified one of the plurality of second depth maps.
11. The method of claim 10 , further comprising forming the first depth map using the video image data with a convolutional neural network (CNN) that performs domain adaptation.
12. The method of claim 11 , further comprising learning the mapping function by the CNN with supervised learning by pairing video image data to corresponding first depth maps, where the first depth maps are formed as a function of the video image data.
13. The method of claim 11 , further comprising learning a mapping function by the CNN with unsupervised generative adversarial learning with a cyclic consistency that does not require paired data, where the first depth map is formed as a function of the video image data.
14. The method of claim 11 , further comprising pre-computing a dataset comprised of paired second depth maps and virtual camera poses, relative to the first depth map.
15. The method of claim 14 , further comprising applying image registration techniques to find a candidate second depth map relative to the dataset with a highest similarity of shape to the first depth map.
16. The method of claim 15 , further comprising using a virtual camera pose value of the candidate second depth map as the camera pose or as an initial value for a candidate virtual camera pose for iteratively generating a next plurality of second depth maps.
17. The method of claim 15 , further comprising estimating a relative pose between the candidate second depth map and the first depth map by passing the candidate second depth map and the first depth map into a spatial transformation network configured to regress a relative transformation between the candidate second depth map and the first depth map.
18. A medical system, comprising:
a medical instrument having a camera associated with a distal end thereof, the camera configured to capture video image data of an internal anatomical structure; and
control circuitry communicatively coupled to the medical instrument, the control circuitry being configured to:
transform the video image data into a first depth map that characterizes the internal anatomical structure based at least in part on a mapping function;
receive a plurality of second depth maps based at least in part on computed tomography (CT) image data, each of the second depth maps representing a virtual model of the internal anatomical structure;
identify one the plurality of second depth maps that has a highest similarity of shape to the first depth map;
estimate camera pose parameters by solving a transformation matrix between a 3 D point cloud of the virtual model of the internal anatomical structure and a point cloud inverse projected from the first depth map; and
generate a camera pose representing a location and an orientation having six-degrees of freedom ( 6 DoF) within the internal anatomical structure, based at least in part on the identified one of the plurality of second depth maps.
19. The medical system of claim 18 , wherein the control circuitry is further configured to represent the first depth map with a first point cloud and a second depth map of the plurality of second depth maps with a second point cloud.
20. The medical system of claim 19 , wherein the control circuitry is further configured to determine deep geometric features based at least in part on the first and second point clouds using a first neural network.
21. The medical system of claim 20 , wherein the control circuitry is further configured to establish a point-wise correspondence between a plurality of key points of the first point cloud and an associated plurality of key points of the second point cloud.
22. The medical system of claim 21 , wherein the control circuitry is further configured to establish the point-wise correspondence by searching in a shared feature space using a second neural network, the shared feature space relative to the deep geometric features.
23. The medical system of claim 21 , wherein the control circuitry is further configured to pass the point-wise correspondence into another network to output a second point-wise correspondence.
24. The medical system of claim 23 , wherein the other network comprises a differentiable random sample consensus (RANSAC) network.
25. The medical system of claim 23 , wherein the control circuitry is further configured to generate the camera pose based at least in part on the second point-wise correspondence.
26. A medical system, comprising:
a means for capturing video image data of an internal anatomical structure;
a means for transforming the video image data into a first depth map that characterizes the internal anatomical structure based at least in part on a mapping function;
a means for receiving a plurality of second depth maps based at least in part on transformed computed tomography (CT) image data, each of the second depth maps representing a virtual model of the internal anatomical structure;
a means for identifying one of the plurality of second depth maps that has a highest similarity of shape to the first depth map; and
a means for generating a camera pose based at least in part on the identified one of the plurality of second depth maps, including estimating camera pose parameters by solving a transformation matrix between a 3D point cloud of the virtual model of the internal anatomical structure and a point cloud inverse projected from the first depth map.
27. The medical system of claim 26 , further comprising an additional means for forming the camera pose using at least one of electromagnetic (EM) data, kinematic data, inertial measurement data, or shape sensing data.
28. The medical system of claim 26 , further comprising a means for communicating the camera pose in real time, relative to the internal anatomical structure.
29. The medical system of claim 28 , wherein the internal anatomical structure is intermittently in motion.