ALIGNING PRE-OPERATIVE SCAN IMAGES TO REAL-TIME OPERATIVE IMAGES FOR A MEDIATED-REALITY VIEW OF A SURGICAL SITE
A mediated-reality system for surgical applications incorporates pre-operative images and real-time captured images of a surgical site into a visualization presented on a head-mounted display worn by a surgeon during a surgical procedure. The mediated-reality system tracks the surgeon's head position and generates real-time images of the surgical site from a virtual camera perspective corresponding to the surgeon's head position to mimic the natural viewpoint of the surgeon. The mediated-reality system furthermore aligns the pre-operative images with the real-time images from the virtual camera perspective and presents a mediated-reality visualization of the surgical site with the aligned pre-operative three-dimensional images or a selected portion thereof overlaid on the real-time images representing the virtual camera perspective.
1 - 20 . (canceled)
21 . A method for generating a mediated reality view of a surgical site, the method comprising:
receiving a pre-operative image representing three-dimensional anatomy of a patient in a first position;
receiving real-time images from a camera array after the patient is positioned for surgery in a second position;
identifying, based on the pre-operative image and the real-time images, a set of features visible in both the pre-operative image and the real-time images;
utilizing an optimization algorithm to identify a transformation that minimizes a distance between first coordinates of the set of features in the pre-operative image with second coordinates of the set of features in the real-time images;
applying the transformation to the pre-operative image to substantially align the pre-operative image with the real-time images;
overlaying the transformed pre-operative image on the real-time images to generate the mediated reality view; and
providing the mediated reality view to a display device for display.
22 . The method of claim 21 wherein the set of features comprises anatomical features of a body of the patient.
23 . The method of claim 21 wherein the set of features comprises fiducial markers coupled to a patch applied to the body of the patient.
24 . The method of claim 23 wherein the fiducial markers comprise three-dimensional structures including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other ones of the fiducial markers.
25 . The method of claim 23 wherein the fiducial markers are arranged in a grid on the patch.
26 . The method of claim 23 wherein the fiducial markers comprise materials of different densities that are distinguishable in the real-time images.
27 . The method of 23 wherein the patch comprises a perforated boundary configured to be torn to enable a first portion of the patch to be removed from the body of the patient while a second portion of the patch remains applied to the body of the patient.
28 . The method of claim 21 wherein the camera array comprises a plurality of cameras fixed in position and orientation relative to one another and configured to capture the real-time images.
29 . The method of claim 28 wherein the cameras comprise light field cameras.
30 . The method of claim 21 wherein the real-time images include encoded depth information.
31 . A device configured to be imaged for aligning a pre-operative image with real-time images of a patient, the device comprising:
an adhesive patch configured to be fixedly applied to a body of a patient; and
a plurality of fiducial markers coupled to the patch, wherein the fiducial markers each comprise a three-dimensional structure including one or more raised surfaces, and wherein each of the fiducial markers has a unique geometry relative to other ones of the fiducial markers,
wherein the patch is configured to be fixedly applied to the body of the patient when the pre-operative image and the real-time images are captured, and
wherein the fiducial markers are visible in the pre-operative image and the real-time images for enabling a registration between the pre-operative image and the real-time images.
32 . The device of claim 31 wherein the fiducial markers are arranged in a grid on the patch.
33 . The device of claim 31 wherein the fiducial markers each comprise a material of different density.
34 . The device of claim 31 wherein the fiducial markers of different densities are distinguishable in ultrasound and/or radiopaque images.
35 . The device of claim 31 wherein the patch comprises a perforated boundary configured to be torn to enable a first portion of the patch to be removed from the patient while a second portion of the patch remains fixedly applied to the body of the patient.
36 . A device configured to be imaged for aligning a pre-operative image with real-time images of a patient, the device comprising:
an adhesive patch configured to be fixedly applied to a body of a patient; and
a plurality of fiducial markers coupled to the patch, wherein the fiducial markers each comprise a material of different density,
wherein the patch is configured to be fixedly applied to the body of the patient when the pre-operative image and the real-time images are captured, and
wherein the fiducial markers are visible in the pre-operative image and the real-time images for enabling a registration between the pre-operative image and the real-time images.
37 . The device of claim 36 wherein the fiducial markers each comprise a three-dimensional structure including one or more raised surfaces.
38 . The device of claim 36 wherein the fiducial markers each comprise a three-dimensional structure including one or more raised surfaces.
39 . The device of claim 36 wherein the fiducial markers are arranged in a grid on the patch.
40 . The device of claim 36 wherein the patch comprises a perforated boundary configured to be torn to enable a first portion of the patch to be removed from the patient while a second portion of the patch remains fixedly applied to the body of the patient.