Systems and methods for reconstruction of 3D images from ultrasound and camera images

1 . A computer-implemented method for reconstructing a 3D image depicting blood flow in a blood vessel of an individual, comprising:

obtaining Doppler ultrasound images depicting a blood vessel with blood flowing therein and measurement of blood flow in a region of the blood vessel, and 2D camera images captured by a camera depicting a plurality of fiducial objects randomly distributed within a viscous material and spaced apart by random distances on a surface of a body segment of the individual,

wherein the Doppler ultrasound images include at least one Doppler ultrasound image, and the 2D camera images include at least one 2D camera image;

computing 3D coordinates within a world coordinate system for pixels of each of the Doppler ultrasound images using an external reference of an ultrasound transducer pose computed by analysis of relative changes in locations of the plurality of fiducial objects within sequential 2D camera images; and

computing a respective estimated blood flow for each of a plurality of pixels of the Doppler ultrasound images at a plurality of locations within the blood vessel;

reconstructing a 3D image from 3D voxels computed from the 3D coordinates of pixels of the Doppler ultrasound images, including respective estimated blood flow,

wherein the 3D image depicts an anatomical image of the blood vessel and depicts blood flow.

2 . The computer-implemented method of claim 1 , wherein the 3D coordinates are further computed by correcting for actual scale to the world coordinate system by using a size of the fiducial object and/or by adding information from other sensors selected from a group consisting of: an inertial measurement unit, and another camera.

3 . The computer-implemented method of claim 1 , wherein the 3D image further comprises voxels computed from a 3D point cloud or mesh depicting the surface of the body segment of the individual.

4 . The computer-implemented method of claim 1 , wherein computing 3D coordinates for pixels of each of the Doppler ultrasound images, comprises:

computing a pose of the camera in a camera coordinate system relative to the world coordinate system by analyzing relative changes in locations of the plurality of fiducial objects within sequential 2D camera images;

applying a calibrated mapping for mapping pixels of the ultrasound images represented in an ultrasound coordinate system to the camera coordinate system, wherein the calibrated mapping is based on a predefined transformation between the pose of the camera and a pose of an ultrasound transducer; and

mapping the pixels of the ultrasound images represented in the camera coordinate system to the 3D coordinates within the world coordinate system.

5 . The computer-implemented method of claim 1 , wherein computing the respective estimated blood flow comprises an estimated blood flow computed from the Doppler ultrasound image captured by the ultrasound transducer located at an angle to a vector denoting a direction of the blood vessel in which the blood flow is measured further comprising correcting the estimated blood flow for the angle to obtain an estimated actual blood flow.

6 . The computer-implemented method of claim 5 , wherein correcting comprises identifying an elliptical-shaped boundary of the blood vessel, computing a transformation of the elliptical shape to a circle, and applying the transformation for projecting the measured blood flow velocity vector to obtain the actual blood flow.

7 . The computer-implemented method of claim 5 , further comprising computing an estimate of a longitudinal axis of the blood vessel based on an aggregation of a plurality of 3D points obtained from respective planes of acquisition of each of a plurality of Doppler ultrasound images, computing a current tangent to the longitudinal axis, and wherein correcting comprises correcting the measurement of blood flow based on an angle between the tangent and a normal to the respective plane of acquisition denoted by the tangent.

8 . The computer-implemented method of claim 5 , further comprising computing an estimate of a blood flow within a precalculated 3D artery object and using computational fluid dynamics simulation and 3D points obtained from respective planes of acquisition of each of a plurality of Doppler ultrasound images for computing an estimate of blood flow within the 3D artery object.

9 . The computer-implemented method of claim 5 , further comprising: for each respective 3D voxel, storing a plurality of initially estimated blood flow values and a corresponding normal to a plane of the ultrasound transducer at which a respective Doppler US image used to compute the respective initial estimated blood flow is captured, estimating actual blood flow in each voxel based on minimizing a projection error between a recorded blood flow velocity vector and an assumed flow velocity vector.

10 . The computer-implemented method of claim 9 , further comprising:

obtaining volumetric flow within the blood vessel on a sagittal view at a specific location for a plurality of cardiac cycles, and computing an average volumetric flow over the plurality of cardiac cycles;

computing a gamma value denoting a real number in a range of −1 to 1 to correct the average volumetric flow and the estimated actual blood flow;

computing a cosine angle by using an inverse of the gamma value; and

correcting the average volumetric flow using the cosine angle and a dot product between estimated blood flow and real blood flow.

11 . The computer-implemented method of claim 1 , further comprising:

alternating between (i) obtaining the Doppler ultrasound images and 2D camera images and (ii) obtaining second images comprising B-mode ultrasound images and 2D camera images;

computing in the world coordinate system, 3D coordinates for pixels of each of the B-mode ultrasound images;

wherein reconstructing the 3D image comprises reconstructing the 3D image from 3D voxels in the world coordinate system computed by aggregating the 3D coordinates of pixels of the B-mode ultrasound images and the Doppler ultrasound images including respective estimated blood flow,

wherein the 3D image depicts the anatomical image of the blood vessel created from the aggregation of 3D voxels obtained from pixels of the B-mode and depicts the blood flow in association with the 3D voxels obtained from pixels of the B-mode.

12 . The computer-implemented method of claim 1 , wherein the respective estimated blood flow is depicted as color coding of the 3D voxels of the 3D image corresponding to the plurality of locations within the blood vessel, wherein pixels of the Doppler ultrasound images are color-coded denoting blood flow, and further comprising segmenting the colored pixels, wherein the 3D image is reconstructed from 3D voxels corresponding to the segmented color pixels.

13 . The computer-implemented method of claim 1 , wherein the respective estimated blood flow of the 3D voxels is selected as maximal values over an imaging time interval during which the Doppler ultrasound images depicting a region within the blood vessel corresponding to the 3D voxels is captured.

14 . The computer-implemented method of claim 1 , wherein a plurality of Doppler ultrasound images used to compute the 3D voxels are captured over an imaging time interval depicting variation in blood flow, and wherein the reconstructed 3D image includes, for the 3D voxels, a respective indication of variation in blood flow over the imaging time interval.

15 . The computer-implemented method of claim 12 , wherein the reconstructed 3D image is presented as a video over an imaging time interval by varying the indication of blood flow corresponding to the 3D voxels over the imaging time interval.