Granted Patent B2

US 12,254,670 · App. 17/877,575 · Granted Mar 18, 2025

Inventors: Souvik Hazra (Munich, DE); Avik Santra (Munich, DE)

Assignee: Infineon Technologies AG

G06V10/764G01S13/89G06V10/82G06V10/84G06V40/23

Abstract

Claims (58)

1. A method comprising:

receiving raw data from a millimeter-wave radar sensor;

generating a first radar-Doppler image based on the raw data;

generating a first radar point cloud based on the first radar-Doppler image;

using a graph encoder to generate a first graph representation vector indicative of one or more relationships between two or more parts of a target based on the first radar point cloud;

generating a first cadence velocity diagram indicative of a periodicity of movement of one or more parts of the target based on the first radar-Doppler image; and

classifying an activity of the target based on the first graph representation vector and the first cadence velocity diagram.

2. The method of claim 1 , further comprising:

generating a plurality of graph representation vectors that corresponds to frames of the raw data that are sequential in time, the plurality of graph representation vectors comprising the first graph representation vector; and

providing the plurality of graph representation vectors to a long short-term memory (LSTM) neural network, wherein classifying the activity of the target is based on an output of the LSTM neural network.

3. The method of claim 2 , further comprising providing the first cadence velocity diagram to a convolutional neural network (CNN) encoder, wherein classifying the activity of the target is based on the CNN encoder.

4. The method of claim 3 , further comprising:

providing an output of the CNN encoder to a deep neural network (DNN);

providing an output of the DNN to a fusion neural network; and

providing the output of the LSTM neural network to the fusion neural network, wherein classifying the activity of the target is based on an output of the fusion neural network.

5. The method of claim 4 , wherein the fusion neural network comprises a fully-connected layer.

6. The method of claim 4 , further comprising training the graph encoder before training the LSTM neural network, the CNN encoder, the DNN, and the fusion neural network.

7. The method of claim 6 , wherein training the graph encoder comprises:

generating a first training camera point cloud based on an output of a camera, wherein the camera has a field of view directed towards a scene, wherein the millimeter-wave radar sensor has a field of view directed towards the scene;

generating a first training radar point cloud based on an output of the millimeter-wave radar sensor, wherein the first training radar point cloud corresponds to the first training camera point cloud;

using the graph encoder to generate a first training graph representation vector based on the first training radar point cloud;

using a graph decoder to generate a second training point cloud based on the first training graph representation vector;

comparing the first training camera point cloud with the second training point cloud to generate error data; and

adjusting one or more weights of the graph encoder based on the error data.

8. The method of claim 1 , wherein the first radar point cloud comprises, for each point of the first radar point cloud, three-dimensional coordinates, an intensity value, and a Doppler value.

9. The method of claim 1 , further comprising generating a first point-angle image based on the first radar-Doppler image, wherein generating the first radar point cloud comprises generating the first radar point cloud based on the first point-angle image.

10. The method of claim 9 , further comprising generating a masked first radar-Doppler image from the first radar-Doppler image based on an intensity of each point of the first radar-Doppler image, wherein generating the first point-angle image comprises generating the first point-angle image based on the masked first radar-Doppler image.

11. The method of claim 10 , wherein generating the masked first radar-Doppler image comprises using a constant false alarm rate (CFAR) detector.

12. The method of claim 10 , wherein generating the masked first radar-Doppler image comprises:

selecting points of the first radar-Doppler image having an intensity higher than a predetermined threshold; and

for each selected point, identifying one or more neighboring points in the first radar-Doppler image, wherein the masked first radar-Doppler image comprises the selected points and the identified neighboring points, and wherein non-selected points and points not identified as neighboring points are not comprised in the masked first radar-Doppler image.

13. The method of claim 9 , wherein generating the first point-angle image comprises performing Barlett beamforming to transform the first radar-Doppler image to a range-angle image.

14. The method of claim 1 , further comprising:

transmitting radar signals towards a scene comprising the target;

receiving reflected radar signals from the scene; and

generating the first radar-Doppler image based on the reflected radar signals.

15. The method of claim 1 , wherein the target is a person, wherein the first graph representation vector is indicative of one or more relationships between two or more body parts of the person, wherein the first cadence velocity diagram is indicative of a periodicity of movement of one or more body parts of the person, and wherein classifying the activity of the target comprises determining whether the person is falling or not falling.

16. The method of claim 1 , wherein the graph encoder is a graph convolutional neural network (CNN) encoder.

17. A radar system comprising:

a radar sensor configured to transmit a plurality of radar signals towards a scene, receive a plurality of reflected radar signals from the scene, and generate raw data based on the plurality of reflected radar signals; and

a processing system configured to:

generate a first radar image based on the raw data,

generate a first radar point cloud based on the first radar image,

use a graph encoder to generate a first graph representation vector indicative of one or more relationships between two or more parts of a target based on the first radar point cloud,

generate a first cadence velocity diagram indicative of a periodicity of movement of one or more parts of the target based on the first radar image, and

classify an activity of the target located in the scene based on the first graph representation vector and the first cadence velocity diagram.

18. The radar system of claim 17 , wherein the processing system is further configured to:

generate a plurality of graph representation vectors that corresponds to frames of the raw data that are sequential in time, the plurality of graph representation vectors comprising the first graph representation vector; and

provide the plurality of graph representation vectors to a recurrent neural network, wherein the processing system is configured to classify the activity of the target based on an output of the recurrent neural network.

19. The radar system of claim 18 , wherein the recurrent neural network comprises a bi-directional long short-term memory (LSTM) neural network.

20. A radar system comprising:

a millimeter-wave radar sensor configured to transmit a plurality of radar signals towards a scene, receive a plurality of reflected radar signals from the scene, and generate raw data based on the plurality of reflected radar signals; and

a processing system configured to:

generate a plurality of radar-Doppler images based on the raw data,

generate a plurality of radar point clouds based on the plurality of radar-Doppler images,

use a graph encoder to generate a plurality of graph representation vectors indicative of one or more relationships between two or more parts of a target based on the plurality of radar point clouds,

generate a first cadence velocity diagram indicative of a periodicity of movement of one or more parts of the target based on the plurality of radar-Doppler images, and

classify an activity of the target located in the scene based on the plurality of graph representation vectors and the first cadence velocity diagram.

Assignments (1)

Continuity (1)

References Cited (215)