Physics-guided deep multimodal embeddings for task-specific data exploitation
A method, apparatus and system for object detection in sensor data having at least two modalities using a common embedding space includes creating first modality vector representations of features of sensor data having a first modality and second modality vector representations of features of sensor data having a second modality, projecting the first and second modality vector representations into the common embedding space such that related embedded modality vectors are closer together in the common embedding space than unrelated modality vectors, combining the projected first and second modality vector representations, and determining a similarity between the combined modality vector representations and respective embedded vector representations of features of objects in the common embedding space to identify at least one object depicted by the captured sensor data. In some instances, data manipulation of the method, apparatus and system can be guided by physics properties of a sensor and/or sensor data.
1. A method for training a common embedding space for combining sensor data captured from a common scene having at least two modalities, the method comprising:
for each of a plurality of the captured sensor data having a first modality of the at least two modalities, creating respective first modality sensor-data vector representations of features of the sensor data having the first modality using a sensor data-specific neural network;
for each of a plurality of the captured sensor data having a second modality of the at least two modalities, creating respective second modality sensor-data vector representations of the features of the sensor data having the second modality using a sensor data-specific neural network;
embedding the first modality sensor-data vector representations and the second modality sensor-data vector representations in a common embedding space such that embedded modality vectors that are related, across modalities, are closer together in the common embedding space than unrelated modality vectors; and
respectively combining the embedded first modality sensor-data vector representations and the second modality vector representations;
wherein at least one of the creating of the first and second modality sensor-data vector representations and the embedding of the first and the second modality sensor-data vector representations are constrained by physics properties of at least one of a respective sensor having captured the first modality sensor data and the second modality sensor data, to prevent captured sensor data having the first modality and sensor data having the second modality captured using physics properties not in compliance with the physics properties of the at least one of the respective sensor from being used to train the common embeddings space, such that the common embedding space is trained using only sensor data having the first modality and sensor data having the second modality captured in compliance with the physics properties of the at least one of the respective sensor.
2. The method of claim 1 , wherein a sensor data-specific neural network is pretrained to recognize features of sensor data having a modality to which the sensor data-specific neural network is to be applied.
3. The method of claim 1 , wherein the first modality sensor-data vector representations and the second modality sensor-data vector representations are combined using late fusion.
4. The method of claim 1 , further comprising:
determining a difference between the plurality of the captured sensor data having the first modality and the captured sensor data having the second modality of the at least two modalities.
5. The method of claim 4 , wherein the determined difference between the captured sensor data having the first modality and the second modality is used to determine missing data of one of the first modality or the second modality from captured data of the other one of the second modality or the first modality.
6. The method of claim 4 , wherein the difference is determined using a generative adversarial network.
7. The method of claim 1 , comprising:
determining a contribution of each of the embedded first modality sensor-data vector representations and the second modality vector representations to the combination.
8. The method of claim 1 , wherein the physics properties comprise at least one of surface reflection, temperature, or humidity.
9. A method for at least one of object detection, object classification, or object segmentation in sensor data having at least two modalities using a common embedding space, comprising:
creating respective first modality sensor-data vector representations of features of sensor data having a first modality of the at least two modalities;
creating respective second modality sensor-data vector representations of features of sensor data having a second modality of the at least two modalities;
projecting the first modality sensor-data vector representations and the second modality sensor-data vector representations into the common embedding space such that embedded modality vectors that are related, across modalities, are closer together in the common embedding space than unrelated modality vectors;
combining the projected first modality sensor-data vector representations and the second modality sensor-data vector representations;
determining a similarity between the combined modality sensor-data vector representations and respective embedded vector representations of features of objects in the common embedding space using a distance function, to identify at least one object depicted by the sensor data having the at least two modalities;
wherein at least one of the creating of the first and second modality sensor-data vector representations and the embedding of the first and the second modality sensor-data vector representations are constrained by physics properties of at least one of a respective sensor having captured the first modality sensor data and the second modality sensor data, to prevent captured sensor data having the first modality and sensor data having the second modality captured using physics properties not in compliance with the physics properties of the at least one of the respective sensor from being used to train the common embeddings space, such that the common embedding space is trained using only sensor data having the first modality and sensor data having the second modality captured in compliance with the physics properties of the at least one of the respective sensor.
10. The method of claim 9 , comprising:
determining a difference between the plurality of the sensor data having the first modality and the sensor data having the second modality of the at least two modalities.
11. The method of claim 10 , wherein at least one of the first modality sensor-data vector representations and the second modality sensor-data vector representations are created using the determined difference between the plurality of the sensor data having the first modality and the sensor data having the second modality.
12. The method of claim 9 , wherein at least one of the first modality sensor-data vector representations and the second modality sensor-data vector representations are created using a sensor data-specific neural network.
13. The method of claim 9 , wherein a contribution of each of the embedded first modality sensor-data vector representations and the second modality vector representations to the combination is predetermined.
14. The method 13 , wherein the first modality sensor-data vector representations and the second modality sensor-data vector representations are combined using attention-based mode fusion.
15. An apparatus for object detection in sensor data having at least two modalities using a common embedding space, comprising:
at least one feature extraction module configured to create respective first modality sensor-data vector representations of features of sensor data having a first modality of the at least two modalities and respective second modality sensor-data vector representations of features of sensor data having a second modality of the at least two modalities;
at least one embedding module configured to project the first modality sensor-data vector representations and the second modality sensor-data vector representations into the common embedding space such that embedded modality vectors that are related, across modalities, are closer together in the common embedding space than unrelated modality vectors;
a fusion module configured to combine the projected first modality sensor-data vector representations and the second modality sensor-data vector representations;
an inference module configured to determine a similarity between the combined modality sensor-data vector representations and respective embedded vector representations of features of objects in the common embedding space using a distance function to identify at least one object depicted by the sensor data having the at least two modalities;
wherein at least one of the creating of the first and second modality sensor-data vector representations and the embedding of the first and the second modality sensor-data vector representations are constrained by physics properties of at least one of a respective sensor having captured the first modality sensor data and the second modality sensor data, to prevent captured sensor data having the first modality and sensor data having the second modality captured using physics properties not in compliance with the physics properties of the at least one of the respective sensor from being used to train the common embeddings space, such that the common embedding space is trained using only sensor data having the first modality and sensor data having the second modality captured in compliance with the physics properties of the at least one of the respective sensor.
16. The apparatus of claim 15 , further comprising:
a generative adversarial network configured to determine a difference between the plurality of the sensor data having the first modality and the sensor data having the second modality of the at least two modalities.
17. The apparatus of claim 16 , wherein the generative adversarial network uses the determined difference between the sensor data having the first modality and the second modality to determine missing data of one of the first modality or the second modality from data of the other one of the second modality or the first modality.
18. The apparatus of claim 15 , wherein the fusion module is configured to determine a contribution of each of the projected first modality sensor-data vector representations and the second modality sensor-data vector representations of the at least two modalities to the combination.
19. The apparatus of claim 18 , wherein the fusion module is configured to apply attention-based mode fusion to combine the first modality sensor-data vector representations and the second modality sensor-data vector representations.
20. The apparatus of claim 15 , wherein the physics properties comprise at least one of surface reflection, temperature, or humidity.