PROCESSING IMAGE DATA
Image data of a first image in a sequence of images is processed using an artificial neural network (ANN) to generate output image data indicative of an alignment of the first image with a second image in the sequence. The ANN is trained using outputs of an alignment pipeline configured to perform alignment of images. The alignment pipeline is configured to determine flow vectors representing optical flow between images, and perform an image transformation using the flow vectors to align the images. The ANN is trained to emulate a result derivable using the alignment pipeline.
1 . A computer-implemented method of processing image data, the method comprising:
receiving image data of a first image in a sequence of images;
processing the received image data using an artificial neural network to generate output image data of the first image, the output image data being indicative of an alignment of the first image with a second image in the sequence of images; and
using the output image data for image processing,
wherein the artificial neural network is trained using outputs of an alignment pipeline configured to perform alignment of images input to the alignment pipeline,
wherein the alignment pipeline is configured to:
determine flow vectors representing optical flow between the images input to the alignment pipeline; and
perform an image transformation using the determined flow vectors to align the images input to the alignment pipeline,
wherein the artificial neural network is trained to emulate a result derivable using the alignment pipeline.
2 . The method according to claim 1 , wherein the image transformation comprises a warping operation for warping at least one of the images input to the alignment pipeline based on the determined flow vectors.
3 . The method according to claim 1 , wherein the first image and the second image are successive images in a temporal sequence of images.
4 . The method according to claim 1 , further comprising using the output image data of the first image to aggregate temporal information of the first image and/or the second image, thereby to enable temporal correlation between the first image and the second image to be used to enhance the first image and/or the second image.
5 . The method according to claim 1 , wherein the output image data of the first image comprises an approximation of a result of performing the image transformation on the image data of the first image using flow vectors representing optical flow between the first image and the second image.
6 . The method according to claim 1 ,
wherein the artificial neural network comprises a series of convolutional filters, and
wherein processing the received image data using the artificial neural network comprises applying the convolutional filters to the received image data.
7 . The method according to claim 6 ,
wherein the image transformation is dependent on content of the images input to the alignment pipeline, and
wherein the convolutional filters of the artificial neural network are independent of content of the first image.
8 . The method according to claim 1 ,
wherein the received image data of the first image comprises a map of image features derivable from the first image, and
wherein the output image data comprises an approximation of a result of aligning the map of image features derivable from the first image with a map of image features derivable from the second image.
9 . The method according to claim 1 , wherein the artificial neural network is trained using a loss function configured to determine a difference between an output of the artificial neural network and the output of the alignment pipeline.
10 . The method according to claim 1 , wherein the alignment pipeline comprises a further artificial neural network trained to determine the flow vectors.
11 . The method according to claim 1 , wherein the artificial neural network comprises a student artificial neural network, and wherein the alignment pipeline comprises a teacher artificial neural network.
12 . The method according to claim 1 ,
wherein the artificial neural network is trained using an affinity distillation loss function configured to determine a difference between a teacher affinity matrix and a student affinity matrix,
wherein the teacher affinity matrix is indicative of dependencies between image features in a map of image features generated by the alignment pipeline, and
wherein the student affinity matrix is indicative of dependencies between image features in a map of image features generated by the artificial neural network.
13 . The method according to claim 1 , further comprising:
receiving image data of the second image;
processing the received image data of the second image using the artificial neural network to generate output image data of the second image, the output image data of the second image being indicative of alignment of the second image with the first image; and
using the output image data of the second image for image processing.
14 . The method according to claim 1 , further comprising concatenating the first image with the second image using the output image data of the first image generated using the artificial neural network.
15 . The method according to claim 1 , further comprising upscaling the first image and/or the second image using the output image data of the first image.
16 . The method according to claim 1 , further comprising denoising the first image and/or the second image using the output image data of the first image.
17 . The method according to claim 1 , wherein the outputs of the alignment pipeline comprise final outputs of the alignment pipeline.
18 . A computer-implemented method of configuring an artificial neural network, the method comprising:
receiving image data of a first image in a sequence of images;
processing the received image data using an artificial neural network to generate output image data of the first image, the output image data indicative of an alignment of the first image with a second image in the sequence of images;
receiving an output of an alignment pipeline configured to perform alignment of images input to the alignment pipeline,
wherein the alignment pipeline is configured to:
determine flow vectors representing optical flow between the images input to the alignment pipeline; and
perform an image transformation using the determined flow vectors to align the images input to the alignment pipeline, and
training the artificial neural network using the output of the alignment pipeline and the output image data of the first image,
wherein the artificial neural network is trained to emulate a result derivable using the alignment pipeline.
19 . A computing device comprising:
a memory comprising computer-executable instructions;
a processor configured to execute the computer-executable instructions and cause the computing device to perform a method of processing image data, the method comprising:
receiving image data of a first image in a sequence of images;
processing the received image data using an artificial neural network to generate output image data of the first image, the output image data being indicative of an alignment of the first image with a second image in the sequence of images; and
using the output image data for image processing,
wherein the artificial neural network is trained using outputs of an alignment pipeline configured to perform alignment of images input to the alignment pipeline,
wherein the alignment pipeline is configured to:
determine flow vectors representing optical flow between the images input to the alignment pipeline; and
perform an image transformation using the determined flow vectors to align the images input to the alignment pipeline,
wherein the artificial neural network is trained to emulate a result derivable using the alignment pipeline.
20 . A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a computing device, cause the computing device to perform a method of processing image data, the method comprising:
receiving image data of a first image in a sequence of images;
processing the received image data using an artificial neural network to generate output image data of the first image, the output image data being indicative of an alignment of the first image with a second image in the sequence of images; and
using the output image data for image processing,
wherein the artificial neural network is trained using outputs of an alignment pipeline configured to perform alignment of images input to the alignment pipeline,
wherein the alignment pipeline is configured to:
determine flow vectors representing optical flow between the images input to the alignment pipeline; and
perform an image transformation using the determined flow vectors to align the images input to the alignment pipeline,
wherein the artificial neural network is trained to emulate a result derivable using the alignment pipeline.