Aligning pre-operative scan images to real-time operative images for a mediated-reality view of a surgical site
A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.
1. A method comprising:
identifying, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery;
applying a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images;
receiving real-time images of the patient captured at least in part during the surgery;
identifying three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient;
applying a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in the real-time images to generate refined transformed images;
overlaying the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and
providing the initial mediated reality view to a display device for display.
2. The method of claim 1 , wherein the set of corresponding features include anatomical features of a body of the patient.
3. The method of claim 1 , wherein the set of corresponding features include fiducial markers on a patch applied to a body of the patient.
4. The method of claim 3 , wherein the patch comprises a flexible adhesive patch positioned proximate to an anatomical structure of interest.
5. The method of claim 3 , wherein the fiducial markers comprise three-dimensional structures including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other fiducial markers on the patch.
6. The method of claim 3 , wherein the fiducial markers are arranged in a grid on the patch.
7. The method of claim 3 , wherein the fiducial markers comprise materials of different densities that are distinguishable in the real-time images.
8. The method of claim 3 , wherein the patch comprises perforated boundaries for tearing the patch into sections.
9. The method of claim 1 , further comprising:
identifying, based on the images of the patient in the first position, coordinates in a three-dimensional image space corresponding to locations of fiducial markers present on a patch applied to the patient; and
identifying, based on the real-time images, coordinates in a three-dimensional real-time image space corresponding to locations of the fiducial markers present on the patch applied to the patient.
10. A non-transitory computer-readable storage medium comprising stored instructions, the instructions comprising instructions for generating a mediated reality view of a surgical site, the instructions causing one or more processors of a mediated reality system to execute processing comprising operations to:
identify, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery;
apply a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images;
receiving real-time images of the patient captured at least in part during the surgery;
identify three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient;
apply a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in the real-time images to generate refined transformed images;
overlay the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and
provide the initial mediated reality view to a display device for display.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the set of corresponding features include anatomical features of a body of the patient.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the set of corresponding features include fiducial markers on a patch applied to a body of the patient.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the patch comprises a flexible adhesive patch positioned proximate to an anatomical structure of interest.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the fiducial markers comprise three-dimensional structures including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other fiducial markers on the patch.
15. The non-transitory computer-readable storage medium of claim 12 , wherein the fiducial markers are arranged in a grid on the patch.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the fiducial markers comprise materials of different densities that are distinguishable in the real-time images.
17. The non-transitory computer-readable storage medium of claim 12 , wherein the patch comprises perforated boundaries for tearing the patch into sections.
18. The non-transitory computer-readable storage medium of claim 10 , further comprising stored instructions that when executed cause the mediated reality system to:
identify, based on the images of the patient in the first position, coordinates in a three-dimensional image space corresponding to locations of fiducial markers present on a patch applied to the patient; and
identify, based on the real-time images, coordinates in a three-dimensional real-time image space corresponding to locations of the fiducial markers present on the patch applied to the patient.
19. A mediated reality system comprising:
a display device;
a processor; and
a non-transitory computer-readable storage medium comprising stored instructions, the instructions comprising instructions for generating a mediated reality view of a surgical site, the instructions when executed by the processor of the mediated reality system, cause the mediated reality system to execute processing comprising operations to:
identify, based on images of a patient in a first position and post-positioning images of the patient in a second position, a set of corresponding features visible in the images of the patient in the first position and the post-positioning images, wherein the images of the patient in the first position are captured prior to a surgery and the post-positioning images are captured after the patient is positioned for the surgery;
apply a first transformation to the images of the patient in the first position to substantially align locations of the corresponding features in the images of the patient in the first position to respective locations in the post-positioning images to generate initial transformed images;
receiving real-time images of the patient captured at least in part during the surgery;
identify three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient;
apply a second transformation to the initial transformed images of the patient in the first position based on the three-dimensional coordinates of features in images of the patient in the first position and three-dimensional coordinates of features in the real-time images of the patient to refine an alignment of the locations of features in the initial transformed images to locations of corresponding features in real-time images to generate refined transformed images;
overlay the refined transformed images over the real-time images of the patient to generate an initial mediated reality view; and
provide the initial mediated reality view to the display device for display.
20. The mediated reality system of claim 19 , wherein the display device comprises a head-mounted display device.