Endoluminal object characterization using 3D-reconstruction
The present disclosure provides for determination of object characteristics/properties based at least in part on images captured from within a subject. The object characteristics/properties can include, without limitations, size, weight, and/or shape of the object. An object depicted in the images may be identified and may be reconstructed in three-dimensional (3D) space as a volumetric representation (e.g., a 3D reconstruction of the object). The object characteristics/properties may be estimated based at least in part on the volumetric representation. In some embodiments, a scale associated with the volumetric representation may be determined and applied to the volumetric representation.
1 . A system for determining a property of an object, the system comprising:
at least one non-transitory computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one non-transitory computer-readable memory and configured to execute the instructions to cause the system to:
access a three-dimensional (3D) representation of the object;
determine a first trajectory of an imaging device over a time period using photometric tracking;
determine a second trajectory of a position sensor over the time period;
determine a scale to be applied to the 3D representation based at least in part on the first trajectory and the second trajectory; and
estimate the property of the object based at least in part on the scale and the 3D representation.
2 . A system for generating a three-dimensional (3D) reconstruction of an object, the system comprising:
an imaging device proximate a distal end of an elongate body, the imaging device configured to capture images of the object;
a robotic system configured to control the elongate body; and
control circuitry configured to:
access image data of the images;
generate depth maps from the image data;
estimate a trajectory of the imaging device over time based at least in part on photometric tracking on the depth maps; and
generate the 3D reconstruction of the object based at least in part on the image data and the trajectory.
3 . A system for determining a property of an object, the system comprising:
at least one non-transitory computer-readable memory having stored thereon executable instructions; and
one or more processors in communication with the at least one non-transitory computer-readable memory and configured to execute the instructions to cause the system to:
access image data of images of the object captured by an imaging device proximate a distal end of an elongate body;
estimatE a trajectory of the imaging device over time based at least in part on identification of one or more features in a first image frame of the image data and corresponding features in a second frame of the image data subsequent to the first image frame;
generate a three-dimensional (3D) reconstruction of the object based at least in part on the image data and the estimating trajectory of the imaging device;
determine a scale to be applied to the 3D reconstruction; and
based at least in part on the scale and the 3D reconstruction, estimate the property ofthe object.
4 . The system of claim 3 , wherein the property is a size property.
5 . The system of claim 3 , wherein the image data comprises image frames, and wherein generating the 3D reconstruction of the object comprises:
generating segmentation masks for the image frames.
6 . The system of claim 3 , wherein the estimating the trajectory of the imaging device comprises: generating a first segmentation mask for the first image frame and a second segmentation mask for the second image frame, the first segmentation mask including the one or more features and the second segmentation mask including the corresponding features; and tracking the one or more features and the corresponding features inside the segmentation masks.
7 . The system of claim 3 , further comprising:
generating depth maps from the image data, the trajectory of the imaging device further estimated based on photometric tracking on the depth maps.
8 . The system of claim 7 , wherein the photometric tracking is based at least in part on keyframe-based photometric error.
9 . The system of claim 3 , wherein generating the 3D reconstruction of the object comprises:
performing a volume fusion based at least in part on at least one of segmentation masks, depth maps, or an estimated trajectory of the imaging device over time; and
generating a first 3D mesh of the object based at least in part on the segmentation masks.
10 . The system of claim 9 , wherein generating the 3D reconstruction of the object further comprises:
generating a second 3D mesh of a scene based at least in part on the estimated trajectory and the depth maps.
11 . The system of claim 10 , wherein the one or more processors is further configured to execute the instructions to cause the system to determine a relative movement between the first 3D mesh of the object and the second 3D mesh of the scene, wherein the relative movement indicates anatomical motion or scope motion.
12 . The system of claim 9 , wherein the volume fusion is based at least in part on at least one of a Truncated Signed Distance Field representation or surfel representation.
13 . The system of claim 3 , further comprising:
accessing electromagnetic (EM) data generated by an EM sensor proximate the distal end of the elongate body; and
determining a transformation between corresponding coordinate frames of the EM sensor and the imaging device.
14 . The system of claim 13 , wherein determining the scale to be applied to the 3D reconstruction comprises:
determining a second trajectory of the EM sensor, the second trajectory in an EM sensor coordinates;
and determining the scale based at least in part on the estimated trajectory, the second trajectory, and the transformation between the EM sensor and the imaging device.
15 . The system of claim 3 , wherein estimating the property of the object comprises:
generating a bounding geometry that encompasses 3D point clouds of the 3D reconstruction; and
multiplying the bounding geometry with the scale to determine object dimensions.
16 . The system of claim 15 , wherein the bounding geometry is an ellipsoid and the object dimensions are represented in minimum volume covering the ellipsoid.
17 . The system of claim 3 , wherein estimating the property of the object comprises:
determining two pixels having the farthest distance apart in an image frame of the image data;
tracking the two pixels in at least one subsequent image frame following the image frame;
performing triangulation to determine a 3D distance corresponding to the two pixels; and
determining the maximum dimension of the object.
18 . The system of claim 3 , further comprising:
estimating poses corresponding to the images based at least in part on a kinematic model and robot data, the 3D reconstruction of the object further generated based on the estimated poses.
19 . The system of claim 3 , wherein the object is at least one of a kidney stone, a medical tool, or an anatomical feature.