Training method, device and image representation system for image stitching
The invention concerns an image processing system configured for carrying out a computer-implemented method for generating a stitched image with a convolutional neural network ( 208 ), a computer-implemented method for generating a stitched image with a convolutional neural network, and a method of training a convolutional neural network, for determining an image representation for image stitching. The training of the convolutional neural network comprises receiving image data, wherein the image data comprises at least two images ( 101, 102 ), overlapping each other in an overlapping area ( 103, 203 ) in which the images overlapping each other in a target anatomy ( 104 ), and wherein the overlapping area comprises an optimum displacement between the two images such that the two images can be correctly combined for image stitching, determining a cost function ( 105, 205 ) of the displacement of the two images in the overlapping area, determining a deviation of the cost function from a reference cost function, and optimizing the CNN based on the deviation.
1 . A method of training a convolutional neural network (CNN) for determining an image representation for image stitching, the method comprising:
receiving image data, wherein the image data comprises at least two images, overlapping each other in an overlapping area in which the images overlap each other in a target anatomy, and wherein the overlapping area comprises an optimum displacement between the two images such that the two images can be correctly combined for image stitching;
determining a cost function of the displacement of the two images in the overlapping area;
determining a deviation of the cost function from a reference cost function; and,
optimizing the CNN based on the deviation.
2 . The method according to claim 1 , further comprising penalizing deviations from the reference cost function.
3 . The method according to claim 1 , wherein determining the cost function, determining the deviation of the cost function from the reference cost function, and optimizing the CNN are repeated until the cost function meets the reference cost function.
4 . The method of training according to claim 1 , wherein the image data is obtained from clinical stitching sequences having annotated correct displacements and/or from artificially constructed image data with known displacement.
5 . The method according to claim 1 , wherein determining the cost function of the displacement of the two images comprises applying a similarity measure to a plurality of displacements between the two images.
6 . The method according to claim 5 , wherein applying the similarity measure comprises determining the similarity of the overlapping area for possible displacements.
7 . The method according to claim 5 , wherein applying the similarity measure comprises generating a cost value for each of the pluralities of displacements, wherein the cost function is generated from the cost values of the plurality of displacements.
8 . The method according to claim 1 , wherein the reference cost function is a convex cost function.
9 . The method according to claim 1 , wherein a minimum of the cost function corresponds to the optimum displacement of the two images.
10 . The method according to claim 1 , wherein the similarity measure comprises at least one of a normalized cross-correlation, a zero mean normalized cross-function, a sum of absolute differences, or a sum of squared differences.
11 . A computer-implemented method for generating a stitched image with a convolutional neural network (CNN), comprising:
receiving an input dataset comprising at least two images, wherein the at least two images comprise an overlapping area in which the images overlap each other in a target anatomy;
processing the input dataset using the CNN trained using a method according to claim 1 ;
producing an output dataset from the CNN comprising at least two output images;
applying a stitching algorithm to the at least two output images of the CNN;
determining optimum displacement using the stitching algorithm; and
generating a composite image, which is a stitched image, based on the determined optimum displacement and the input images.
12 . The method according to claim 11 , wherein processing the input data set comprises applying at least one trained filter kernel.
13 . An image processing system for generating a stitched image with a convolutional neural network (CNN), the system comprising:
at least one processor configured to
receive an input dataset comprising at least two images, wherein the at least two images comprise an overlapping area in which the images overlap each other in a target anatomy;
process the input dataset using the CNN trained using a method according to claim 1 ;
produce an output dataset from the CNN comprising at least two output images;
apply a stitching algorithm to the at least two output images of the CNN;
determine an optimum displacement using the stitching algorithm; and
generate a composite image, which is a stitched image, based on the determined optimum displacement and the input images.
14 . A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 1 .