Registration of 3D and 2D images for surgical navigation and robotic guidance without using radiopaque fiducials in the images

1 . A system for registering a 3D image volume comprising:

a portable imaging system having an imaging tracking marker array (ITMA);

a dynamic reference base (DRB) including patient tracking markers and attachable to a patient;

a tracking device configured to track a position and orientation of both the ITMA and the DRB simultaneously;

a processor adapted to:

receive first and second 2D images at different orientations that have been captured intra-operatively by the imaging system;

receive corresponding optical images of the patient from the tracking device at a time of image capture, the optical images containing the ITMA and the DRB;

register the first and second 2D images to the patient using the captured images from the imaging system and the optical images containing the ITMA and the DRB;

generate a customized 3D image volume from a generic 3D image volume based on the first and second 2D images;

merge the 2D images to the customized 3D image volume so as to register the customized 3D image volume to the patient based on a registration information of the 2D images,

wherein the processor is configured to register the first 2D image to the patient by:

determining transform A representing a pose of the imaging system in an imaging space relative to the DRB in a physical space at the time the first 2D image is captured based on the received optical images.

2 . The system of claim 1 , wherein the processor is configured to register the first 2D image to the patient by:

determining transform B representing a pose of the first 2D image in an imaging space relative to the imaging system in the physical space based on the received optical images.

3 . The system of claim 2 , wherein the processor is configured to determine the transform B by determining a spatial relationship between the first 2D image and the ITMA which has been adjusted with calibration data related to an amount of flex of an imaging arm at different orientation of the imaging system.

4 . The system of claim 2 , the processor is configured to determine the transform B by determining a position of a detector panel relative to a position of the ITMA.

5 . The system of claim 1 , wherein the processor is configured to display the registered customized 3D image volume on a display and superimpose a planned trajectory of a surgical instrument, and dynamically adjust the displayed trajectory over the displayed 3D image volume as the displayed trajectory is varied by a user based on the registration information of the customized 3D image volume.

6 . The system of claim 1 , further comprising a surgical robot configured to assist in a surgical operation of the patient and be in communication with the imaging system, the surgical robot including a base, a robot arm coupled to the base, an end effector coupled to the robot arm and a robot processor, wherein:

under control of the imaging system, the processor is configured to:

attach the registration information in an image file of the first and second 2D images; and

transfer the image file containing the 2D images and the registration information from the imaging system to the surgical robot; and

wherein the robot processor is configured to merge the 2D images to the customized 3D image volume based on the registration information contained in the image file.

7 . The system of claim 1 , wherein the processor is configured to receive intra-op ultrasound images as the 2D images.

8 . The system of claim 1 , wherein the processor is configured to register the first 2D image to the patient by:

determining transform B representing a pose of the first 2D image in the imaging space relative to the imaging system in the physical space based on the received optical images; and

determining transform C by multiplying the transform A with the transform B, the transform C representing a registration of the pose of the first 2D image relative to the DRB in the physical space.

9 . The system of claim 8 , wherein the processor is configured to determine the transform A by:

generating a transform A1 representing a pose of the DRB relative to the cameras of the tracking device;

generating a transform A2 representing the pose of the ITMA relative to cameras of the tracking device;

multiplying the transform A1 and the transform A2 together.

10 . The system of claim 9 , wherein the processor generates an inverse of the transform A2 prior to multiplying.

11 . The system of claim 8 , wherein the processor is configured to determine the transform B by:

generating a transform B1 representing a pose of a detector relative to the ITMA of the imaging system;

generating a transform B2 representing a pose of a medical images in the imaging space relative to the detector;

multiplying the transform B1 and the transform B2 together.

12 . The system of claim 8 , wherein the processor is configured to determine the transform B by determining a spatial relationship between the 2D images and the ITMA which has been adjusted with calibration data related to an amount of flex at different orientations of the imaging system.

13 . The system of claim 1 , wherein the processor is configured to segment the first 2D image prior to generating a customized 3D image volume.

14 . A system for registering a 3D image volume comprising:

a portable imaging system having an imaging tracking marker array (ITMA);

a dynamic reference base (DRB) including patient tracking markers and attachable to a patient;

a tracking device configured to track a position and orientation of both the ITMA and the DRB simultaneously;

a surgical robot configured to assist in a surgical operation of the patient and be in communication with the imaging system, the surgical robot including a base, a robot arm coupled to the base and an end effector coupled to the robot arm;

a processor adapted to:

receive first and second 2D images at different orientations that have been captured intra-operatively by the imaging system;

receive corresponding optical images of the patient from the tracking device at a time of image capture, the optical images containing the ITMA and the DRB;

register the first and second 2D images to the patient using the captured images from the imaging system and the optical images containing the ITMA and the DRB;

generate a customized 3D image volume from a generic 3D image volume based on the first and second 2D images;

merge the 2D images to the customized 3D image volume so as to register the customized 3D image volume to the patient based on a registration information of the 2D images; and

move the end effector to a position for receiving a surgical instrument along a planned trajectory based on the registration information of the customized 3D image volume,

wherein the processor is configured to register the first 2D image to the patient by:

determining transform B representing a pose of the first 2D image in an imaging space relative to the imaging system in a physical space based on the received optical images.

15 . The system of claim 14 , wherein the processor is configured to segment the first and second 2D images prior to generating a customized 3D image volume.

16 . The system of claim 14 , wherein the processor is configured to register the first 2D image to the patient by:

determining transform A representing a pose of the imaging system in an imaging space relative to the DRB in the physical space at the time the first 2D image is captured based on the received optical images.

17 . The system of claim 14 , wherein the processor is configured to determine the transform B by determining a spatial relationship between the first 2D image and the ITMA which has been adjusted with calibration data related to an amount of flex of an imaging arm at different orientation of the imaging system.

18 . The system of claim 14 , the processor is configured to determine the transform B by determining a position of a detector panel relative to a position of the ITMA.