Detection and classification of traffic signs using camera-radar fusion

1 . A system comprising:

a sensing system of a vehicle, the sensing system configured to acquire:

a set of camera images of an environment, and

a set of radar images of the environment;

a data processing system of the vehicle, the data processing system configured to:

generate, using a first neural network (NN), a set of camera features characterizing the set of camera images, each camera feature of the set of camera features generated for a respective pixel of a plurality of pixels of a coordinate system and for a respective time of a plurality of times;

generate, using a second NN, a set of radar features characterizing the set of radar images, each radar feature of the set of radar features generated for said respective pixel of the plurality of pixels of the coordinate system and said respective time of the plurality of times;

fuse the set of camera features with the set of radar features to obtain a fused tensor, where the fused tensor aggregates the set of camera features with the set of radar features (i) across pixels of the coordinate system and (ii) across multiple times of the plurality of times; and

process, using a third NN, the fused tensor, wherein an output of the third NN comprises semantic content of one or more traffic signs in the environment; and a driving control system of the vehicle, configured to:

control the vehicle based on the semantic content of the one or more traffic signs.

2 . The system of claim 1 , wherein the coordinate system is associated with a ground surface, and wherein to generate the set of camera features, the data processing system is configured to:

map the set of camera features from a perspective coordinate system to the coordinate system.

3 . The system of claim 1 , wherein the third NN comprises a backbone NN and one or more classification NNs.

4 . The system of claim 1 , wherein the data processing system is further configured to:

process, using a fourth NN, at least a subset of the set of camera features to obtain an auxiliary identification of at least one traffic sign in the environment; and

use the auxiliary identification as an additional input into the third NN.

5 . The system of claim 1 , wherein the first NN, the second NN, and the third NN are trained together.

6 . The system of claim 1 , wherein the semantic content of an individual traffic sign of the one or more traffic signs comprises a determination of one or more of:

a type of the individual traffic sign,

a value associated with a semantic content of the individual traffic sign, or

a relevance of the individual traffic sign for the vehicle; and

wherein the output of the third NN further comprises:

a location of the individual traffic sign.

7 . The system of claim 1 , wherein the data processing system is further configured to:

identify one or more duplicate traffic signs in the one or more traffic signs; and

eliminate the one or more duplicate traffic signs.

8 . A method comprising:

obtaining, using a sensing system of a vehicle:

a set of camera images of an environment, and

a set of radar images of the environment;

generating, using a first neural network (NN), a set of camera features characterizing the set of camera images, each camera feature of the set of camera features generated for a respective pixel of a plurality of pixels of a coordinate system and for a respective time of a plurality of times,

generating, using a second NN, a set of radar features characterizing the set of radar images, each radar feature of the set of radar features generated for said respective pixel of the plurality of pixels of the coordinate system and said respective time of the plurality of times;

fusing the set of camera features with the set of radar features to obtain a fused tensor, where the fused tensor aggregates the set of camera features with the set of radar features (i) across pixels of the coordinate system and (ii) across multiple times of the plurality of times; and

processing, using a third NN, the fused tensor, wherein an output of the third NN comprises semantic content of one or more traffic signs in the environment; and

controlling the vehicle based on the semantic content of the one or more traffic signs.

9 . The method of claim 8 , wherein the coordinate system is associated with a ground surface, and wherein generating the one or more camera features comprises:

mapping the set of camera features from a perspective coordinate system to the coordinate system.

10 . The method of claim 8 , wherein the third NN comprises a backbone NN and one or more classification NNs.

11 . The method of claim 8 , further comprising:

processing, using a fourth NN, at least a subset of the set of camera features to obtain an auxiliary identification of at least one traffic sign in the environment; and

using the auxiliary identification as an additional input into the third NN.

12 . The method of claim 8 , wherein the semantic content of an individual traffic sign of the one or more traffic signs comprises a determination of one or more of:

a type of the individual traffic sign,

a value associated with a semantic content of the individual traffic sign, or

a relevance of the individual traffic sign for the vehicle; and

wherein the output of the third NN further comprises:

a location of the individual traffic sign.

13 . The method of claim 8 , further comprising:

identifying one or more duplicate traffic signs among the one or more traffic signs; and

eliminating the one or more duplicate traffic signs.

14 . An autonomous vehicle comprising:

one or more cameras configured to acquire a set of camera images of an environment;

one or more radar sensors configured to acquire a set of radar images of the environment;

a data processing system of the vehicle, the data processing system configured to:

generate, using a first neural network (NN), a set of camera features characterizing the set of camera images, each camera feature of the set of camera features generated for a respective pixel of a plurality of pixels of a coordinate system and for a respective time of a plurality of times;

generate, using a second NN, a set of radar features characterizing the set of radar images, each radar feature of the set of radar features generated for said respective pixel of the plurality of pixels of the coordinate system and said respective time of the plurality of times;

fuse the set of camera features with the set of radar features to obtain a fused tensor, where the fused tensor aggregates the set of camera features with the set of radar features (i) across pixels of the coordinate system and (ii) across multiple times of the plurality of times; and

process, using a third NN, the fused tensor, wherein an output of the third NN comprises semantic content of one or more traffic signs in the environment; and an autonomous vehicle control system configured to:

cause the autonomous vehicle to follow a driving path selected in view of the semantic content of the one or more traffic signs.