Computer-implemented method for registering low dimensional images with a high dimensional image, a method for training an aritificial neural network useful in finding landmarks in low dimensional images, a computer program and a system for registering low dimensional images with a high dimensional image
A computer-implemented method for registering low dimensional images with a high dimensional image includes receiving a high dimensional image of a region of interest and simulating synthetic low dimensional images of the region of interest from a number of poses of a virtual low dimensional imaging device, from the high dimensional image. The method determines positions of landmarks within the low dimensional images by applying a first learning algorithm to the low dimensional images and back projecting of the positions of the determined landmarks into the high dimensional image space, to thereby obtain the positions of the landmarks in the high dimensional image. The positions of landmarks within low dimensional images acquired form an imaging device are determined by applying the first or a second learning algorithm to the low dimensional images. The low dimensional images are registered with the high dimensional image based on the positions of the landmarks.
1. A computer-implemented method for registering low dimensional images with a high dimensional image, the method comprising the steps of:
a) receiving a high dimensional image of a region of interest,
b) simulating synthetic low dimensional images of the region of interest from a number of poses of a virtual low dimensional imaging device, from the high dimensional image,
c) determining positions of landmarks within the synthetic low dimensional images by applying a first learning algorithm to the low dimensional images,
d) back projection of the positions of the determined landmarks into the high dimensional image space, to thereby obtain the positions of the landmarks determined in step c) in the high dimensional image,
e) receiving low dimensional images acquired with a low dimensional imaging device of the region of interest,
f) determining positions of landmarks within the low dimensional images by applying the first or second learning algorithm to the low dimensional images, and
g) registering the low dimensional images with the high dimensional image based on the positions of the landmarks.
2. The method of claim 1 ,
wherein the step of registering includes a step of determining the pose of the low dimensional imaging device used to acquire the low dimensional images, and
wherein in the step on determining the pose, a perspective-n-point problem is solved, which has the positions of the landmarks within the low dimensional images and the positions of the landmarks within the high dimensional image as an input and the pose of the low dimensional imaging device as an output.
3. The method of claim 1 ,
wherein the first learning algorithm is trained by the input training date and by output training data,
wherein the input training data comprises synthetic low dimensional images determined by simulating synthetic low dimensional images of the region of interest from a number of poses of a virtual low dimensional imaging device, from an arbitrary high dimensional image including the region of interest, and
wherein the output training date comprises positions of landmarks in the low dimensional images and is generating by manual landmark definition in the high dimensional image and projecting the landmarks from the high dimensional image to the low dimensional images.
4. The method of claim 1 ,
wherein the second learning algorithm is trained by input training data and by output training data,
wherein the input training data comprises synthetic low dimensional images determined in step b), and
wherein the output data comprises positions of the landmarks in the low dimensional images, generated by
determining landmarks within the synthetic low dimensional images by applying the first learning algorithm to the synthetic low dimensional images,
back projecting each of the determined landmarks of the synthetic low dimensional images in a hyperplane on the high dimensional space, wherein each hyperplane extends from each of the determined landmarks to the origin of the synthetic low dimensional image,
generating new landmarks within the high dimensional image based on the hyperplanes, and
projecting the new landmarks of the high dimensional image to the synthetic low dimensional images.
5. The method of claim 4 , wherein each new landmark is generated at the intersection of the hyperplanes relating to the same landmark, within the high dimensional image.
6. The method of claim 4 ,
wherein the hyperplanes include projection lines extending through one landmark and the origin of the synthetic low dimensional image respectively, and
wherein each new landmark is generated by determining for each possible pair of projecting lines the point which has the smallest Euclidean distance to both lines, and by determining the barycenter of all points determined from all pairs of projection lines.
7. The method of claim 1 , wherein the second learning algorithm is generated by retraining the first learning algorithm.
8. The method of claim 1 , wherein the first learning algorithm and the second learning algorithm are a first trained artificial neural network and a second trained artificial neural network.
9. The method of claim 1 , wherein the landmarks are anatomical features.
10. The method of claim 1 , wherein the low dimensional image is one of the following: a 2 Dimensional (2D) X-ray image, a 2 Dimensional (2D) ultrasound image, a 2 Dimensional (2D) optical image, a 2 Dimensional (2D) scintigraphy or 2 Dimensional (2D) optoacoustic image.
11. The method of claim 1 , wherein the high dimensional image is one of the following: a 3 Dimensional (3D) Computed Tomography image, 3 Dimensional (3D) Magnetic Resonance image, a 3 Dimensional (3D) ultrasound image, a 3 Dimensional (3D) Nuclear medicine image, a 3 Dimensional (3D) depth camera image, or a 3 Dimensional (3D) optoacoustic image.
12. A method for training or retraining an artificial neural network useful in finding landmarks in low dimensional images, the method comprising:
receiving input training date, namely synthetic low dimensional images of a region of interest simulated from a number of poses of a virtual low dimensional imaging device, from a high dimensional image of the region of interest,
receiving output training data, namely synthetic low dimensional images of the region of interest and position of landmarks within the low dimensional images,
wherein the output training data comprises positions of the landmarks in the low dimensional images, generated by
determining landmarks within the synthetic low dimensional images by applying a first artificial neural network to the synthetic low dimensional image,
back projecting each of the determined landmarks of the synthetic low dimensional images in a hyperplane on the high dimensional space, wherein each hyperplane extends from each of the determined landmarks to the origin of the synthetic low dimensional image,
generating new landmarks within the high dimensional image bases on the hyperplanes, and
projecting the new landmarks of the high dimensional image to the synthetic low dimensional images.
13. A system for registering low dimensional images with a high dimensional image, the system comprising:
a first interface, configured for receiving a high dimensional image,
a second interface, configured for receiving low dimensional images from an imaging device during a surgery,
a computational unit, configured for executing the method according to claim 1 , and
a third interface, configured for outputting a registered image.