Combined surface reconstruction and registration for laparoscopic surgery

1. A computer-implemented method of performing a combined surface reconstruction and registration of stereo laparoscopic images, the method comprising:

generating, by a computer system, an intraoperative three-dimensional model of an anatomical area of interest;

receiving, by the computer system, a plurality of stereo endoscopic images of the anatomical area of interest from a laparoscope;

performing, by the computer system, a first stereo reconstruction process using the plurality of stereo endoscopic images to yield a first surface image corresponding to the anatomical area of interest;

aligning, by the computer system, the first surface image with a surface of the a pre-operative three-dimensional model using a rigid registration process;

performing, by the computer system, a second stereo reconstruction process using the first surface image and prior knowledge from the pre-operative three-dimensional model to yield a second surface image corresponding to the anatomical area of interest; and

aligning, by the computer system, the second surface image with the surface of the intraoperative three-dimensional model using a non-rigid registration process.

2. The method of claim 1 , wherein the intraoperative three-dimensional model is generated using a rotational angiography system.

3. The method of claim 2 , wherein generating the intraoperative three-dimensional model of the anatomical area of interest comprises:

acquiring a group of rotational projection images via the rotational angiography system;

reconstructing a plurality of two-dimensional images based on the group of rotational projection images; and

generating the intraoperative three-dimensional model based on the reconstructed two-dimensional images.

4. The method of claim 2 , wherein at least one of the first stereo reconstruction process and the second stereo reconstruction process comprises:

identifying one or more specular highlights in the plurality of stereo endoscopic images; and

increasing image density of the plurality of stereo endoscopic images based on the one or more specular highlights.

5. The method of claim 4 , wherein at least one of the rigid registration process and the non-rigid registration process utilizes the one or more specular highlights in aligning a respective surface image with the surface of the intraoperative three-dimensional model.

6. The method of claim 1 , wherein at least one of the first stereo reconstruction process and the second stereo reconstruction process comprises:

constructing a graph based on the plurality of stereo endoscopic images, wherein each vertex in the graph corresponds to a voxel in the plurality of stereo endoscopic images and each vertex is connected to a predetermined number of its nearest neighbors by edges;

determining a cost for each edge of the graph;

applying a weighting to the cost associated with each edge using data from the prior knowledge; and

identifying a minimum set of voxels as a respective surface image based on the weighted cost associated with each edge.

7. The method of claim 6 , wherein the predetermined number is 6.

8. The method of claim 6 , wherein each edge is associated with a first direction, a second direction, or a third direction and wherein determining the cost to each edge of the graph comprises:

for each edge associated with the first direction, applying a first cost to each respective edge based on a matching strength value between patches in the plurality of stereo endoscopic images; and

for each edge in the second or the third direction, applying a second cost to each respective edge based on a smoothness enforcement value associated with at least one of the first stereo reconstruction process and the second stereo reconstruction process.

9. A computer-implemented method of performing a combined surface reconstruction and registration of stereo laparoscopic images, the method comprising:

generating an intraoperative three-dimensional model of an anatomical area of interest;

receiving a plurality of stereo endoscopic images of the anatomical area of interest from a laparoscope; and

performing, by a computer, an iterative process a plurality of times until a registration error value is below a threshold value, the iterative process comprising:

performing a stereo reconstruction process using the plurality of stereo endoscopic images to yield a surface image corresponding to the anatomical area of interest,

performing a registration process to align the surface image with a surface of the intraoperative three-dimensional model, and

updating the registration error value based on a displacement of one or more points of the surface image resulting from the registration process.

10. The method of claim 9 , wherein generating the intraoperative three-dimensional model of the anatomical area of interest comprises:

acquiring a group of rotational projection images via a rotational angiography system;

reconstructing a plurality of two-dimensional images based on the group of rotational projection images; and

generating the intraoperative three-dimensional model based on the reconstructed two-dimensional images.

11. The method of claim 9 , wherein the stereo reconstruction process comprises:

identifying one or more specular highlights in the plurality of stereo endoscopic images;

increasing image density of the plurality of stereo endoscopic images based on the one or more specular highlights.

12. The method of claim 11 , wherein the registration process utilizes the one or more specular highlights to align the surface image with the surface of the intraoperative three-dimensional model.

13. The method of claim 11 , wherein performing the stereo reconstruction process using the plurality of stereo endoscopic images to yield the surface image comprises:

constructing a graph based on the plurality of stereo endoscopic images, wherein each vertex in the graph corresponds to a voxel in the plurality of stereo endoscopic images and each vertex is connected to a predetermined number of its nearest neighbors by edges;

determining a cost for each edge of the graph;

applying a weighting to the cost associated with each edge based on prior knowledge data; and

identifying a minimum set of voxels as the surface image based on the weighted cost associated with each edge.

14. The method of claim 13 , wherein each edge is associated with a first direction, a second direction, or a third direction and wherein determining the cost to each edge of the graph comprises:

for each edge associated with the first direction, applying an first cost to each respective edge based on a matching strength value between patches in the plurality of stereo endoscopic images; and

for each edge in the second or the third direction, applying a second cost to each respective edge based on a smoothness enforcement value associated with the stereo reconstruction process.

15. The method of claim 13 , wherein the prior knowledge data includes one or more of the intraoperative three-dimensional model, a pre-operative three-dimensional model, sparse stereo reconstruction data, or manual annotation data.

16. A system for performing a combined surface reconstruction and registration of stereo laparoscopic images during a surgical operation, the system comprising:

an intraoperative imaging system configured to generate an intraoperative three-dimensional model of an anatomical area of interest;

a receiver module configured to receive a plurality of stereo endoscopic images of the anatomical area of interest from a laparoscope; and

an imaging computer configured to perform an iterative process a plurality of times until a registration error value is below a threshold value, the iterative process comprising:

performing a stereo reconstruction process using the plurality of stereo endoscopic images to yield a surface image corresponding to the anatomical area of interest,

performing a registration process to align the surface image with a surface of the intraoperative three-dimensional model, and

updating the registration error value based on a displacement of one or more points of the surface image resulting from the registration process.

17. The system of claim 16 , wherein the intraoperative imaging system comprises a rotational angiography system.

18. The system of claim 17 , further comprising a display configured to present the aligned surface image during the surgical operation.

19. The system of claim 17 , further comprising a tracking system configured to provide tracking data to the imaging computer for use in an initial registration of the intraoperative imaging system to the laparoscope.

20. The system of claim 19 , wherein the tracking system is an optical tracking system.