Methods, systems, and computer readable media for deriving a three-dimensional (3D) textured surface from endoscopic video

1. A method for deriving a three-dimensional (3D) textured surface from endoscopic video, the method comprising:

performing video frame preprocessing to identify a plurality of video frames of an endoscopic video, wherein the video frame preprocessing includes informative frame selection, specularity removal, and key-frame selection;

generating, using a neural network or a shape-from-motion-and-shading (SfMS) approach, a three-dimensional (3D) textured surface using the plurality of video frames; and

registering the 3D textured surface to at least one computed tomography (CT) image, wherein registering the 3D textured surface to the at least one CT image comprises identifying a tissue-gas surface based on the at least one CT image and performing a surface-to-surface registration between the 3D textured surface and the tissue-gas surface.

2. The method of claim 1 comprising:

marking, via a user interface, a region of interest (ROI) of the 3D textured surface, wherein after registering the 3D textured surface, the ROI is visible on the at least one CT image.

3. The method of claim 1 wherein generating the 3D textured surface comprises:

generating a partial 3D surface for each of the plurality of video frames using the SfMS approach; and

fusing, using a group-wise surface registration algorithm and a seamless texture fusion approach, the partial 3D surfaces into the 3D textured surface.

4. The method of claim 1 wherein the SfMS approach comprises:

estimating camera motion and 3D scene structure associated with an input image using a structure-from-motion (SfM) approach or a depth estimation model, wherein the 3D scene structure is parameterized by a SfM point cloud; and

generating, using an iterative update scheme, a new estimated 3D surface for the input image after each iteration, wherein the iterative update scheme stops when convergence is reached or another condition is met.

5. The method of claim 4 wherein the depth estimation model utilizes a recurrent neural network.

6. The method of claim 4 wherein the iterative update scheme comprises: during an iteration:

warping an estimated surface to the SfM point cloud associated with the input image;

estimating a reflectance model using this warped surface and the input image; and

producing, using the estimated surface and the reflectance model, a new estimated surface using a regularized structure-from-shading (SfS) equation, wherein a subsequent iteration uses the new estimated surface when warping.

7. The method of claim 3 wherein the seamless texture fusion approach comprises:

generating an initial texture for the 3D textured surface by, for each voxel on the 3D textured surface, selecting an image whose partial 3D surface has the closest distance to that voxel to color it; and

performing texture fusion by minimizing within-patch and inter-patch differences using an augmented Lagrangian method.

8. The method of claim 1 wherein performing the surface-to-surface registration comprises:

using a thin shell demons (TSD) registration algorithm and applying anisotropic elasticity parameters estimated in a group-wise surface registration to deform the 3D textured surface; and

using an expectation-maximization algorithm to align compatible surface regions between the deformed 3D textured surface and the tissue-gas surface.

9. A system for deriving a three-dimensional (3D) textured surface from endoscopic video, the system comprising:

at least one processor; and

an endoscopogram generation (EG) module executable by the at least one processor, wherein the EG module is configured for:

performing video frame preprocessing to identify a plurality of video frames of an endoscopic video, wherein the video frame preprocessing includes informative frame selection, specularity removal, and key-frame selection;

generating, using a neural network or a shape-from-motion-and-shading (SfMS) approach, a three-dimensional (3D) textured surface using the plurality of video frames; and

registering the 3D textured surface to at least one computed tomography (CT) image, wherein registering the 3D textured surface to the at least one CT image comprises identifying a tissue-gas surface based on the at least one CT image and performing a surface-to-surface registration between the 3D textured surface and the tissue-gas surface.

10. The system of claim 9 wherein the EG module is configured for:

marking, via a user interface, a region of interest (ROI) of the 3D textured surface, wherein after registering the 3D textured surface, the ROI is visible on the at least one CT image.

11. The system of claim 9 wherein generating the 3D textured surface comprises:

generating a partial 3D surface for each of the plurality of video frames using the SfMS approach; and

fusing, using a group-wise surface registration algorithm and a seamless texture fusion approach, the partial 3D surfaces into the 3D textured surface.

12. The system of claim 9 wherein the SfMS approach comprises:

estimating camera motion and 3D scene structure associated with an input image using a structure-from-motion (SfM) approach or a depth estimation model, wherein the scene structure is parameterized by a SfM point cloud; and

generating, using an iterative update scheme, a new estimated 3D surface for the input image after each iteration, wherein the iterative update scheme stops when convergence is reached or another condition is met.

13. The system of claim 12 wherein the depth estimation model utilizes a recurrent neural network.

14. The system of claim 12 wherein the iterative update scheme comprises: during an iteration:

warping an estimated surface to the SfM point cloud associated with the input image;

estimating a reflectance model using this warped surface and the input image; and

producing, using the estimated surface and the reflectance model, a new estimated surface using a regularized structure-from-shading (SfS) equation, wherein a subsequent iteration uses the new estimated surface when warping.

15. The system of claim 11 wherein the seamless texture fusion approach comprises:

generating an initial texture for the 3D textured surface by, for each voxel on the 3D textured surface, selecting an image whose partial 3D surface has the closest distance to that voxel to color it; and

performing texture fusion by minimizing within-patch and inter-patch differences using an augmented Lagrangian method.

16. The system of claim 9 wherein performing the surface-to-surface registration comprises:

using a thin shell demons (TSD) registration algorithm and applying anisotropic elasticity parameters estimated in a group-wise surface registration to deform the 3D textured surface; and

using an expectation-maximization algorithm to align compatible surface regions between the deformed 3D textured surface and the tissue-gas surface.

17. A non-transitory computer readable medium having stored thereon executable instructions that when executed by a processor of a computer cause the computer to perform steps comprising:

performing video frame preprocessing to identify a plurality of video frames of an endoscopic video, wherein the video frame preprocessing includes informative frame selection, specularity removal, and key-frame selection;

generating, using a neural network or a shape-from-motion-and-shading (SfMS) approach, a three-dimensional (3D) textured surface using the plurality of video frames; and

registering the 3D textured surface to at least one computed tomography (CT) image, wherein registering the 3D textured surface to the at least one CT image comprises identifying a tissue-gas surface based on the at least one CT image and performing a surface-to-surface registration between the 3D textured surface and the tissue-gas surface.

18. The non-transitory computer readable medium of claim 17 wherein the executable instructions when executed by the processor of the computer cause the computer to perform steps comprising:

marking, via a user interface, a region of interest (ROI) of the 3D textured surface, wherein after registering the 3D textured surface, the ROI is visible on the at least one CT image.