PREDICTING VISIBLE/INFRARED BAND IMAGES USING RADAR REFLECTANCE/BACKSCATTER IMAGES OF A TERRESTRIAL REGION
The present invention relates to a method and apparatus that can predict the visible-infrared band images of a region of the Earth's surface that would be observed by an Earth Observation (EO) satellite or other high-altitude imaging platform, using data from radar reflectance/backscatter of the same region. The method and apparatus can be used to predict images of the Earth's surface in the visible-infrared bands when the view between an imaging instrument and the ground is obscured by cloud or some other medium that is opaque to electromagnetic (EM) radiation in the visible-infrared spectral range, approximately spanning 400-2300 nanometres (nm), but transparent to EM radiation in the radio-/microwave part of the spectrum. Regular, uninterrupted monitoring of the Earth's surface is important for a wide range of applications, from agriculture to defence.
1 . A method of creating a mapping model for translating an input image to an output image, the method comprising obtaining an ensemble of training data T comprising a sample of pairs of matched images [R,V], providing a neural network and training the neural network with the training data T to obtain the mapping model V*=f(R) that translates input image R to output image V*, where V* is equivalent to the ground truth V in a flawless mapping the method further comprising the following steps:
a) propagating R into the generator, wherein the generator produces V* which represents a “fake” version of V based on a transformation of R
b) associating V* with R to form new matched pair [R,V*]
c) propagating [R,V*] into the discriminator to determine the probability that V* is “real”, wherein the probability that V* is “real” is estimated from a loss function that encodes the quantitative distance between V and V*
d) backpropagating the error defined by the loss function through the neural network wherein R comprises at least one image in a first frequency range, encoded as a data matrix and wherein V comprises at least one image in a second frequency range, encoded as a data matrix.
2 . A method according to claim 1 wherein the training data T comprises a plurality of real matched images [R,V].
3 . A method according to claim 1 wherein the neural network comprises a generator and a discriminator.
4 . A method according to claim 1 wherein there are N iterations of training steps a to d wherein T is sampled at each iteration.
5 . A method according to claim 1 wherein the loss function is learnt by the neural network.
6 . A method according to claim 1 wherein the loss function is hard-coded.
7 . A method according to claim 1 wherein the loss function is a combination of hard-coding and learning by the neural network.
8 . A method according to claim 7 wherein the loss function is a combination of a learnt GAN loss, and a Least Absolute Deviations (L1) loss, with the L1 loss weighted at a fraction of the GAN loss.
9 . A method according to claim 1 wherein each image in R and V is normalised.
10 . A method according to claim 9 wherein normalisation comprises a rescaling of the input values to floating point values in a fixed range.
11 . A method according to claim 1 wherein the neural network comprises an encoder-decoder neural network.
12 . A method according to claim 1 wherein the neural network comprises a conditional GAN.
13 . A method according to claim 1 wherein the neural network comprises a fully convolutional conditional GAN.
14 . A method according to claim 1 wherein the backpropagation of the error defined by the loss function updates the weights in the neural network so that they follow the steepest descent of the loss between V and V*.
15 . A method according to claim 1 wherein R comprises at least one SAR image, encoded as a data matrix.
16 . A method according to claim 1 wherein V comprises at least one image in the visible-infrared spectral range, encoded as a data matrix.
17 . A method according to claim 1 wherein the visible-infrared spectral range is between about 400-2300 nanometres (nm).
18 . A method according to claim 1 wherein R is of size m×n of a patch of the Earth's surface spanning a physical region p×q.
19 . A method according to claim 1 wherein V is of size m×n of a patch of the Earth's surface spanning a physical region p×q.
20 . A method according to claim 1 wherein V is of size m×n at one or more frequencies across the visible-infrared spectral range.
21 . A method according to claim 1 wherein V* is of size m×n of a patch of the Earth's surface spanning a physical region p×q.
22 . A method according to claim 1 wherein V* is of size m×n at one or more frequencies across the visible-infrared spectral range.
23 . A method according to claim 1 wherein where there are a plurality of images R they are all recorded at a single radar frequency.
24 . A method according to claim 1 wherein where there are a plurality of images R they are recorded at multiple frequencies.
25 . A method according to claim 1 wherein where there are a plurality of images R they are all recorded at a single polarisation.
26 . A method according to claim 1 wherein where there are a plurality of images R they are recorded at multiple polarisations.
27 . A method according to claim 1 wherein where there are a plurality of images R they are recorded at different detection orientations/incident angles.
28 . A method according to claim 1 wherein R further comprises additional information representing prior knowledge about the region of interest or the observing conditions of V and/or R.
29 . A method according to claim 28 wherein the additional information includes but is not limited to a map of the surface elevation; a previously observed unobscured view in each visible-infrared spectral band; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.
30 . A method according to claim 28 wherein the additional information is selected from one or more of: a map of the surface elevation; a previously recorded unobscured view in each visible-infrared spectral band; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.
31 . An imaging apparatus for creating a mapping model for translating an input image to an output image using the method according to claim 1 .
32 . A method of translating an input image R to an output image V*, the method comprising obtaining a mapping model for translating an input image to an output image according to claim 1 inputting a new image R into the mapping model wherein the mapping model translates input image R and outputs image V*.
33 . A method according to claim 32 wherein the input R comprises at least one SAR image, encoded as a data matrix.
34 . A method according to claim 32 wherein the output V* comprises at least one image in the visible-infrared spectral range, encoded as a data matrix.
35 . A method according to claim 32 wherein the visible-infrared spectral range is between about 400-2300 nanometres (nm).
36 . A method according to claim 32 wherein the input image R is of size m×n.
37 . A method according to claim 32 wherein the input image R is of size m×n of a patch of the Earth's surface spanning a physical region p×q.
38 . A method according to claim 32 wherein the output image V* is of size m×n of a patch of the Earth's surface spanning a physical region p×q.
39 . A method according to claim 32 wherein the output image V* is of size m×n at one or more frequencies across the visible-infrared spectral range.
40 . A method according to claim 32 wherein where there are a plurality of input images R they are all recorded at a single radar frequency.
41 . A method according to claim 32 wherein where there are a plurality of input images R they are recorded at multiple frequencies.
42 . A method according to claim 32 wherein where there are a plurality of input images R they are all recorded at a single polarisation.
43 . A method according to claim 32 wherein where there are a plurality of input images R they are recorded at multiple polarisations.
44 . A method according to claim 32 wherein where there are a plurality of input images R they are recorded at different detection orientations/incident angles.
45 . A method according to claim 32 wherein R further comprises additional information representing prior knowledge about the region of interest or the observing conditions.
46 . A method according to claim 45 wherein the additional information includes but is not limited to a map of the surface elevation; a previously observed unobscured view in each visible-infrared spectral band; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.
47 . A method according to claim 45 wherein the additional information is selected from one or more of: a map of the surface elevation; a previously observed unobscured view in each visible-infrared spectral band; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.
48 . A method of predicting the visible-infrared band images of a region of the Earth's surface that would be observed by an EO satellite or other high-altitude imaging platform, using data from SAR imaging of the same region using the method of claim 32 .
49 . A method according to claim 48 used to predict images of the Earth's surface in the visible-infrared bands when the view between an imaging instrument and the ground is obscured by cloud or some other medium that is opaque to EM radiation in the visible-infrared spectral range, spanning approximately 400-2300 nanometres (nm), but transparent to EM radiation in the radio-/microwave part of the spectrum.
50 . An imaging apparatus for translating an input image R to an output image V* according to claim 32 .
51 . A method according to claim 32 further comprising generating a new set of images V+ at any frequency in the range approximately spanning 400-2300 nm from V*.
52 . A method as claimed in claim 51 comprising the following steps:
a) considering a pixel at coordinate (x,y) in each image in V*, wherein V* can be considered a set of images V*=[V0, V1, V2, . . . . VN] wherein each image corresponds to an observed bandpass at some average wavelength of EM radiation and wherein the set of wavelengths associated with each image is lambda=[lambda0, lambda1,lambda2 . . . lambdaN];
b) assuming a function S(x,y,lambda,p) represents the continuous spectral response of the Earth surface, where p are a set of parameters. S is described by Equation 1 and p represents 6 free parameters;
c) finding p for each pixel (x,y) by fitting the function S(x,y,lambda,p) to (lambda,V*)
d) creating a new set of images V+ covering the same region as V* by applying S(x,y,lambda,p) for any given wavelength lambda
S
(
λ
)
=
[
p
0
(
1
+
exp
(
-
p
1
(
λ
-
p
2
)
)
)
-
1
+
p
3
]
×
exp
(
-
p
4
(
λ
/
1500
nm
)
)
+
p
5
exp
(
-
(
λ
-
c
)
2
/
2
g
2
)
.
Equation
1