Granted Patent B2

US 12,411,228 · App. 17/661,223 · Granted Sep 9, 2025

Inventors: Kanishka Tyagi (Agoura Hills, CA); Yihang Zhang (Calabasas, CA); Kaveh Ahmadi (Simi Valley, CA); Shan Zhang (Los Angeles, CA); Narbik Manukian (Los Angeles, CA)

Assignee: APTIV TECHNOLOGIES AG

G01S13/89G06T3/4046G06T3/4053

Abstract

Claims (81)

1. A method comprising: obtaining, from an electromagnetic sensor, sensor data; generating, based only on the sensor data, a first sensor image representing the sensor data in multiple dimensions; generating, based only on the sensor data, a second sensor image having a higher resolution in at least one of the multiple dimensions than the first sensor image; and training, by machine learning and based on the first sensor image being used as input data and the second sensor image being used as ground truth data, a model to generate a high-resolution sensor image that approximates the second sensor image, the generated high-resolution sensor image used for at least one of detecting objects, tracking objects, classification, or segmentation.

2. The method of claim 1 , wherein the model being trained comprises at least one artificial neural network.

3. The method of claim 2 , wherein the model further comprises one or more down-sampling layers and one or more up-sampling layers.

4. The method of claim 2 , wherein at least one of the at least one artificial neural network is a convolutional neural network including one or more down-sampling layers, one or more up-sampling layers, and one or more convolution layers.

5. The method of claim 4 , wherein a rectangular filter kernel, configured to compensate for a lower resolution in one of the multiple dimensions, is used to train the model.

6. The method of claim 4 , wherein the model comprises:

an input layer configured to receive the first sensor image as input data and the second sensor image as ground truth data;

a first convolution layer configured to receive an output of the input layer as input;

a first down-sampling layer configured to receive an output of the first convolution layer as input;

a second convolution layer configured to receive an output of the first down-sampling layer as input;

a second down-sampling layer configured to receive an output of the second convolution layer as input;

a first up-sampling layer configured to receive an output of the second down-sampling layer as input;

a third convolution layer configured to receive an output of the first up-sampling layer as input;

a second up-sampling layer configured to receive an output of the third convolution layer as input;

a fourth convolution layer configured to receive an output of the second up-sampling layer as input; and

an output layer configured to receive an output of the fourth convolution layer as input and output the generated high-resolution sensor image.

7. The method of claim 1 , wherein:

the input data is based on absolute values of sensor data magnitudes in the multiple dimensions; and

the absolute values of the sensor data magnitudes are input into the model via a single input channel.

8. The method of claim 1 , wherein the electromagnetic sensor is a radar system.

9. The method of claim 8 , wherein the multiple dimensions comprise at least two of:

a range domain;

a Doppler domain;

an azimuth-angle domain; or

an elevation-angle domain.

10. The method of claim 9 , wherein generating the first sensor image comprises:

determining, based on the sensor data, a set of beam vectors that encompasses all range bins and Doppler bins associated with the sensor data; and

selecting a subset of beam vectors, including a set of Doppler bins at all range bins, from the beam vectors that encompass all the range bins and Doppler bins; and

obtaining, based on the subset of beam vectors, a range-angle map.

11. The method of claim 10 , wherein:

the set of beam vectors are derived from data received by sparse receive channels and dense receive channels of the radar sensor;

the input data is based on magnitudes and phases of the subset of beam vectors received by the sparse and the dense receive channels;

the magnitudes of the beam vectors received by the sparse and the dense receive channels are input on a first channel of the model;

the phases of the beam vectors received by the sparse and the dense receive channels are input on a second channel of the model; and

training the model further comprises using a deep neural network and a filter kernel configured to capture features of the input data.

12. The method of claim 10 , further comprising:

using a Discrete Fourier Transform by applying a Fast Fourier Transform (FFT) to the subset of beam vectors;

using a filter size approximating the input data, and wherein:

the set of beam vectors are derived from data received by sparse receive channels and dense receive channels of the radar sensor;

the input data is based on magnitudes and phases of the subset of beam vectors received by the sparse and the dense receive channels;

the magnitudes of the beam vectors received by the sparse receive channels are input on a first channel of the model;

the phases of the beam vectors received by the sparse receive channels are input on a second channel of the model;

the magnitudes of the beam vectors received by the dense receive channels are input on a third channel of the model; and

the phases of the beam vectors received by the dense receive channels are input on a fourth channel of the model.

13. The method of claim 12 , wherein the magnitudes of the subset of beam vectors are bounded.

14. The method of claim 1 , wherein the model is trained using a general adversarial network.

15. The method of claim 1 , wherein training the model comprises:

comparing the generated high-resolution sensor image to the second sensor image;

calculating, based on comparing the generated high-resolution sensor image to the second sensor image, an error for the generated high-resolution sensor image; and

further training the model based on the error.

16. The method of claim 1 , wherein the second sensor image is generated based on an iterative adaptive algorithm.

17. A system comprising:

a radar sensor;

a trained model trained to generate a high-resolution sensor image;

at least one processor configured to:

receive, from the radar sensor, radar data;

determine, based only on the radar data, a set of beam vectors derived from the radar data that encompasses all range bins and Doppler bins associated with the sensor data;

select a subset of the beam vectors, including a set of Doppler bins at all range bins, from the beam vectors that encompass all the range bins and Doppler bins;

apply a Fast Fourier Transform (FFT) to the subset of beam vectors;

input, into the trained model, magnitudes of the subset of beam vectors into at least a first channel of the model;

input, into the trained model, phases of the subset of beam vectors into at least a second channel of the model;

receive, from the trained model, the generated high-resolution sensor image; and

output the generated high-resolution sensor image for at least one of detecting objects, tracking objects, classification, or segmentation.

18. The system of claim 17 , wherein the trained model comprises:

an input layer;

at least one convolution layer;

at least one down-sampling layer;

at least one up-sampling layer; and

an output layer.

19. The system of claim 17 , wherein the model comprises:

an input layer configured to receive the magnitudes of the subset of beam vectors on at least the first channel of the model and the phases of the subset of beam vectors on at least the second channel of the model;

a first convolution layer configured to receive an output of the input layer as input;

a first down-sampling layer configured to receive an output of the first convolution layer as input;

a second convolution layer configured to receive an output of the first down-sampling layer as input;

a second down-sampling layer configured to receive an output of the second convolution layer as input;

a first up-sampling layer configured to receive an output of the second down-sampling layer as input;

a third convolution layer configured to receive an output of the first up-sampling layer as input;

a second up-sampling layer configured to receive an output of the third convolution layer as input;

a fourth convolution layer configured to receive an output of the second up-sampling layer as input; and

an output layer configured to receive an output of the fourth convolution layer as input and output the generated high-resolution sensor image.

20. The system of claim 17 , wherein the trained model is trained using a general adversarial network.

Assignments (4)

Continuity (2)

References Cited (11)