Predicting visible/infrared band images using radar reflectance/backscatter images of a terrestrial region

1. A method of predicting visible-infrared band images of a region of the Earth's surface that would be observed by an EO satellite or other high-altitude imaging platform, using data from SAR imaging of the same region, the method comprising:

obtaining a mapping model for translating the input image to the output image, wherein the method of obtaining the mapping model comprises:

i) obtaining an ensemble of training data T comprising a sample of pairs of matched images [R,V],

ii) providing a neural network and training the neural network with the training data T to obtain the mapping model V*=f(R) that translates input image R to output image V* where V* is equivalent to V in a flawless mapping,

wherein R comprises at least one SAR image of the region of the Earth's surface, encoded as a data matrix and wherein R is of size m×n of a patch of the Earth's surface spanning a physical region p×q, wherein V comprises at least one image, encoded as a data matrix and wherein V is of size m×n of a patch of the Earth's surface spanning a physical region p×q, wherein R further comprises additional information representing prior knowledge about the region of interest or the observing conditions of V and/or R, and wherein the additional information includes but is not limited to one or more of: a map of the surface elevation; a previously observed unobscured view in one or more visible-infrared spectral bands; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information;

inputting a new image R into the mapping model;

translating, by the mapping model, input image R; and

outputting, by the mapping model, image V*, wherein the output V* comprises at least one visible-infrared band image of the region of the Earth's surface, encoded as a data matrix and wherein the output V* is of size m×n of a patch of the Earth's surface spanning a physical region p×q.

2. The method according to claim 1 wherein the mapping model is used to predict the visible-infrared band images of the region of the Earth's surface when the view between an imaging instrument and the ground is obscured by cloud or some other medium that is opaque to EM radiation in the visible-infrared spectral range, spanning approximately 400-2300 nanometres (nm), but transparent to EM radiation in the radio-/microwave part of the spectrum.

3. The method according to claim 1 wherein the additional information is selected from one or more of: a map of the surface elevation; a previously recorded unobscured view in one or more visible-infrared spectral bands; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.

4. A method of translating an input image R to an output image V*, the method comprising:

a) obtaining a mapping model for translating the input image to the output image, wherein the method of obtaining the mapping model comprises:

i) obtaining an ensemble of training data T comprising a sample of pairs of matched images [R,V],

ii) providing a neural network and training the neural network with the training data T to obtain the mapping model V*=f(R) that translates input image R to output image V* where V* is equivalent to V in a flawless mapping,

wherein R comprises at least one image, encoded as a data matrix and wherein R is of size m×n of a patch of the Earth's surface spanning a physical region p×q, wherein V comprises at least one image, encoded as a data matrix and wherein V is of size m×n of a patch of the Earth's surface spanning a physical region p×q, wherein R further comprises additional information representing prior knowledge about the region of interest or the observing conditions of V and/or R wherein the additional information includes but is not limited to one or more of: a map of the surface elevation; a previously observed unobscured view in one or more visible-infrared spectral bands; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information; and

b) inputting a new image R into the mapping model;

wherein the mapping model translates input image R and outputs image V* wherein the output V* comprises at least one image, encoded as a data matrix and wherein the output V* is of size m×n of a patch of the Earth's surface spanning a physical region p×q.

5. A method according to claim 4 wherein the neural network comprises a generator and a discriminator.

6. A method according to claim 5 wherein the method of obtaining the mapping model comprises the following steps:

5) propagating R into the generator, wherein the generator produces V* which represents a “fake” version of V based on a transformation of R

6) associating V* with R to form new matched pair [R,V*]

7) propagating [R,V*] into the discriminator to determine the probability that V* is “real”, wherein the probability that V* is “real” is estimated from a loss function that encodes the quantitative distance between V and V*

8) backpropagating the error defined by the loss function through the neural network.

7. A method according to claim 6 wherein there are N iterations of training steps 1 to 4 wherein T is sampled at each iteration.

8. A method according to claim 6 wherein the loss function is learnt by the neural network, or

wherein the loss function is hard-coded, or

wherein the loss function is a combination of hard-coding and learning by the neural network.

9. A method according to claim 8 wherein the loss function is a combination of a learnt GAN loss, and a Least Absolute Deviations (L1) loss, with the L1 loss weighted at a fraction of the GAN loss.

10. A method according to claim 4 wherein the neural network comprises an encoder-decoder neural network.

11. A method according to claim 4 wherein the neural network comprises a conditional GAN.

12. A method according to claim 4 wherein the neural network comprises a fully convolutional conditional GAN.

13. A method according to claim 6 wherein the backpropagation of the error defined by the loss function updates the weights in the neural network so that they follow the steepest descent of the loss between V and V*.

14. A method according to claim 4 wherein R comprises at least one SAR image, encoded as a data matrix,

wherein V comprises at least one image in the visible-infrared spectral range, encoded as a data matrix, and

wherein V* comprises at least one image in the visible-infrared spectral range, encoded as a data matrix.

15. A method according to claim 14 wherein V is of size m×n at one or more frequencies across the visible-infrared spectral range, and

wherein V* is of size m×n at one or more frequencies across the visible-infrared spectral range.

16. A method according to claim 4 wherein where there are a plurality of images R they are all recorded at a single radar frequency.

17. A method according to claim 4 wherein where there are a plurality of images R they are recorded at multiple frequencies.

18. A method according to claim 4 wherein where there are a plurality of images R they are all recorded at a single polarisation.

19. A method according to claim 4 wherein where there are a plurality of images R they are recorded at multiple polarisations.

20. A method according to claim 4 wherein where there are a plurality of images R they are recorded at different detection orientations/incident angles.

21. A method according to claim 4 wherein the additional information is selected from one or more of: a map of the surface elevation; a previously recorded unobscured view in one or more visible-infrared spectral bands; a map of the location of each pixel; time of year; and sun elevation/azimuth angle information.

22. A method according to claim 4 further comprising predicting the visible-infrared band images of a region of the Earth's surface that would be observed by an EO satellite or other high-altitude imaging platform, using data from SAR imaging of the same region, wherein output V* comprises the predicted visible-infrared band images.

23. A method according to claim 22 wherein the visible-infrared band images of the Earth's surface are predicted when the view between an imaging instrument and the ground is obscured by cloud or some other medium that is opaque to EM radiation in the visible-infrared spectral range, spanning approximately 400-2300 nanometres (nm), but transparent to EM radiation in the radio-/microwave part of the spectrum.

24. An imaging apparatus for translating an input image R to an output image V* according to claim 4 .

25. A method according to claim 4 further comprising generating a new set of images V+ at any frequency in the range approximately spanning 400-2300 nm from V*.

26. A method as claimed in claim 25 comprising the following steps:

e) considering a pixel at coordinate (x,y) in each image in V*, wherein V* can be considered a set of images V*=[V0, V1, V2, . . . VN] wherein each image corresponds to an observed bandpass at some average wavelength of EM radiation and wherein the set of wavelengths associated with each image is lambda=[lambda0, lambda1, lambda2 . . . lambdaN];

f) assuming a function S(x,y,lambda,p) represents the continuous spectral response of the Earth surface, where p are a set of parameters, S is described by Equation 1, and p represents 6 free parameters;

g) finding p for each pixel (x,y) by fitting the function S(x,y,lambda,p) to (lambda,V*); and

h) creating a new set of images V+ covering the same region as V* by applying S(x,y,lambda,p) for any given wavelength lambda

S (λ)=[ p 0 (1+exp(− p 1 (λ− p 2 ))) −1 +p 3 ]×exp(− p 4 (λ/1500 nm))+ p 5 exp(−(λ− c ) 2 /2 g 2 )   Equation 1.