Defending multimodal fusion models against single-source adversaries

1. A multimodal perception system comprising:

a controller configured to,

receive a first signal from a first sensor, a second signal from a second sensor, and a third signal from a third sensor,

extract a first feature vector from the first signal,

extract a second feature vector from the second signal,

extract a third feature vector from the third signal,

determine an odd-one-out vector from the first, second, and third feature vectors via an odd-one-out network of a machine learning network, based on inconsistent modality prediction,

fuse, via a fusion network, the first, second, and third feature vectors and odd-one-out vector into a fused feature vector, and

output the fused feature vector.

2. The multimodal perception system of claim 1 , wherein the first sensor, second sensor, and third sensor are each of a different modality.

3. The multimodal perception system of claim 2 , wherein the odd-one-out vector is of length modalities plus 1, and each modality has a perturbation with the plus 1 being indicative of an unperturbed modality.

4. The multimodal perception system of claim 3 , wherein the controller determines the odd-one-out network and fuses the feature vectors with the odd-one-out vector as a Convolutional Neural Networks (CNN) to align spatiotemporal dimensions of the different modalities.

5. The multimodal perception system of claim 1 , wherein the controller is further configures to extract the first feature vectors from the first signal, via a first pre-trained AI model, extract the second feature vectors from the second signal, via a second pre-trained AI model, extract the third feature vectors from the third signal, via a third pre-trained AI model.

6. The multimodal perception system of claim 5 , wherein the controller is further configured to jointly train the odd-one-out network according to a loss function expressed by

-

𝔼

(

x

,

y

)

∼

𝒟

z

i

=

g

i

(

x

i

)

[

log

⁢

o

⁡

(

z

)

k

+

1

+

∑

i

=

1

k

⁢

log

⁢

o

⁡

(

z

i

*

,

z

-

i

)

i

]

,

in parallel with a task modality, in which z* i =g i (x* i )is a feature extracted from perturbed input x* i , and o is the odd-one-out network, and in response to a stopping criteria event, halt the joint training.

7. The multimodal perception system of claim 1 , wherein the controller is further configures to fuse the first, second, and third feature vectors and out-one-out vector into a fused feature vector according to

e i ( z )= NN (⊕ z −i )∀ i ∈[ k ], e k+1 ( z )= NN (⊕ z ),

in which ⊕ denotes a concatenation operation, NN stands for a shallow neural network, z is input, k is a modality, e i is a fusion of features from all the modalities except for i, and only e k+1 fuses features from all the modalities.

8. The multimodal perception system of claim 1 , wherein the first sensor is one of video, RADAR, LIDAR, or ultrasound, and the controller is further configured to control an autonomous vehicle based on the fused feature vector.

9. The multimodal perception system of claim 1 , wherein the first sensor is one of video, sound, IR, or LIDAR, and the controller is further configured to control an access door based on the fused feature vector.

10. The multimodal perception system of claim 1 , wherein the first sensor is one of video, sound, ultrasound, IR, or LIDAR, and the controller is further configured to control a mechanical system.

11. A multimodal perception method comprising:

receiving a first signal from a first sensor, a second signal from a second sensor, and a third signal from a third sensor;

extracting a first feature vectors from the first signal, a second feature vectors from the second signal, and a third feature vectors from the third signal;

determining an odd-one-out, vector from the first, second, and third feature vectors via an odd-one-out network of a machine learning network based on an inconsistent modality prediction;

fusing, via a fusion network, the first, second, and third feature vectors and odd-one-out vector into a fused feature vector; and

outputting the fused feature vector.

12. The multimodal perception method of claim 11 wherein the first sensor, second sensor, and third sensor are each of a different modality.

13. The multimodal perception method of claim 12 , wherein the odd-one-out vector is of length modalities plus 1, and each modality has a perturbation with the plus 1 being indicative of an unperturbed modality.

14. The multimodal perception method of claim 13 , wherein determining the odd-one-out network and fusing the feature vectors with the odd-one-out vector is via a Convolutional Neural Networks (CNN) to align spatiotemporal dimensions of the different modalities.

15. The multimodal perception method of claim 11 , wherein extracting the first feature vectors from the first signal, via a first pre-trained AI model, extracting the second feature vectors from the second signal, via a second pre-trained Al model, extracting the third feature vectors from the third signal, via a third pre-trained Al model.

16. The multimodal perception method of claim 15 further comprising, jointly training the odd-one-out network according to a loss function expressed by

-

𝔼

(

x

,

y

)

∼

𝒟

z

i

=

g

i

(

x

i

)

[

log

⁢

o

⁡

(

z

)

k

+

1

+

∑

i

=

1

k

⁢

log

⁢

o

⁡

(

z

i

*

,

z

-

i

)

i

]

,

in parallel with a task modality, in which z* i =g i (x* i ) is a feature extracted from perturbed input x* i , and o is the odd-one-out network, and in response to a stopping criteria event, halt the joint training.

17. The multimodal perception method of claim 11 , wherein fusing the first, second, and third feature vectors and out-one-out vector into a fused feature vector according to

e i ( z )= NN (⊕ z −i )∀ i ∈[ k ], e k+1 ( z )= NN ( ⊕z ),

in which ⊕ denotes a concatenation operation, NN stands for a shallow neural network, z is input, k is a modality, e i is a fusion of features from all the modalities except for i, and only e k+1 fuses features from all the modalities.

18. A multimodal perception system for an autonomous vehicle comprising:

a first sensor that is one of a video, RADAR, LIDAR, or ultrasound sensor; and

a controller configured to,

receive a first signal from a first sensor, a second signal from a second sensor, and a third signal from a third sensor,

extract a first feature vector from the first signal,

extract a second feature vector from the second signal,

extract a third feature vector from the third signal,

determine an odd-one-out vector from the first, second, and third feature vectors via an odd-one-out network of a machine learning network, based on inconsistent modality prediction,

fuse, via a fusion network, the first, second, and third feature vectors and odd-one-out vector into a fused feature vector,

output the fused feature vector, and

control the autonomous vehicle based on the fused feature vector.

19. The multimodal perception system of claim 18 , wherein the first sensor, second sensor, and third sensor are each of a different modality, and wherein the odd-one-out vector is of length modalities plus 1, and each modality has a perturbation with the plus 1 being indicative of an unperturbed modality.

20. The multimodal perception system of claim 19 , wherein the controller determines the odd-one-out network and fuses the feature vectors with the odd-one-out vector as a Convolutional Neural Networks (CNN) to align spatiotemporal dimensions of the different modalities.