Machine learned registration and multi-modal regression
Systems, devices, methods, and computer-readable media for image processing by machine learning are provided. A method can include providing, as input to a machine learning (ML) model for image processing, data of a first image, receiving, from the ML model, an estimated radar cross section (RCS) of the first image, receiving, from the ML model, estimated transformation parameters that project the first image to a second image space, and applying the estimated transformation parameters to the estimated RCS to generate an estimated second image in the second image space.
1. A system for image registration and multi-modal regression, the system comprising:
a memory to store parameters defining a machine learning (ML) model for image processing; and
processing circuitry to:
provide, as input to the ML model, data of a first image;
receive, from a fully-convolutional layer of the ML model, an estimated radar cross section (RCS) of the first image, the RCS is an effective area that scatters incident signals back to a radar and governed by a monostatic radar equation;
receive, from a convolutional residual network or densely-connected convolution network of the ML model, estimated transformation parameters that project the first image to a second image space of a synthetic aperture radar (SAR) or synthetic aperture sonar (SAS) image; and
apply the estimated transformation parameters to the estimated RCS to generate an estimated second image in the second image space, the estimated second image is an estimated SAR or SAS image.
2. The system of claim 1 , wherein the processing circuitry is further to, provide a second image of the geographical region to the ML model.
3. The system of claim 2 , wherein the first image is generated by a first type of image sensor and the second image is generated by a second type of image sensor.
4. The system of claim 3 , wherein the first type includes one of a visible spectrum sensor, an infrared sensor, and a light detection and ranging sensor and the second type includes one of a synthetic aperture radar and a synthetic aperture sonar sensor.
5. The system of claim 3 , wherein:
the input to the first portion includes the first image; and
the input to the second portion, during training, includes the first image and the second image.
6. The system of claim 5 , wherein the first portion is further to estimate depth in the first image and the processing circuitry is further to apply the estimated depth, along with the transformation parameters to the RCS to generate the estimated second image in the second image space.
7. The system of claim 1 , wherein the ML model is trained using first images generated using a first sensor of a first sensor type and second images generated using a second sensor of a different second sensor type, and feedback that is a difference between the second image and the estimated second image.
8. The system of claim 1 , wherein the processing circuitry is to apply the estimated transform to the first image to register the first and second images to each other.
9. A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
providing, as input to a machine learning (ML) model for image processing, data of a first image;
receiving, from a fully-convolutional layer of the ML model; an estimated radar cross section (RCS) of the first image, the RCS is an effective area that scatters incident signals back to a radar and governed by a monostatic radar equation;
receiving, from a convolutional residual network or densely-connected convolution network of the ML model, estimated transformation parameters that project the first image to a second image space of a synthetic aperture radar (SAR) or synthetic aperture sonar (SAS) image; and
applying the estimated transformation parameters to the estimated RCS to generate an estimated second image in the second image space, the estimated second image is an estimated SAR or SAS image.
10. The non-transitory machine-readable medium of claim 9 , wherein the operations further comprise providing a second image to the ML model, the second image including an overlapping field of view with the first image.
11. The non-transitory machine-readable medium of claim 10 , wherein the first image is generated by a first type of image sensor and the second image is generated by a second type of image sensor.
12. The non-transitory machine-readable medium of claim 11 , wherein the first type includes one of a visible spectrum sensor, an infrared sensor, and a light detection and ranging sensor and the second type includes one of a synthetic aperture radar and a synthetic aperture sonar sensor.
13. The non-transitory machine-readable medium of claim 11 , wherein:
the input to the first portion includes the first image; and
the input to the second portion, during training, includes the first image and the second image.
14. The non-transitory machine-readable medium of claim 13 , wherein the first portion is further to estimate depth in the first image and the processing circuitry is further to apply the estimated depth, along with the transformation parameters to the RCS to generate the estimated second image in the second image space.
15. A method for multi-modal regression performed by a machine learning (ML) technique implemented using processing circuitry of a machine, the method comprising:
providing, as input to the ML technique, data of a first image;
receiving, from a fully-convolutional layer of the ML technique, an estimated radar cross section (RCS) of the first image, the RCS is an effective area that scatters incident signals back to a radar and governed by a monostatic radar equation;
receiving, from a convolutional residual network or densely-connected convolution network of the ML technique, estimated transformation parameters that project the first image to a second image space of a synthetic aperture radar (SAR) or synthetic aperture sonar (SAS) image: and
applying the estimated transformation parameters to the estimated RCS to generate an estimated second image in the second image space, the estimated second image is an estimated SAR or SAS image.
16. The method of claim 15 , further comprising training the ML technique using first images generated using a sensor of a first sensor type and second images generated using a sensor of a different second sensor type, and feedback that is a difference between the second image and the estimated second image.
17. The method of claim 15 , further comprising applying the estimated transform to the first image to register the first and second images to each other.
18. The method of claim 15 , wherein:
the input to the first portion includes the first image;
the input to the second portion, during training, includes the first image and the second image; and
the first portion is further to estimate depth in the first image and the processing circuitry is further to apply the estimated depth, along with the transformation parameters to the RCS to generate the estimated second image in the second image space.