Data fusion and analysis engine for vehicle sensors
Systems and methods for data fusion and analysis of vehicle sensor data, including receiving a multiple modality input data stream from a plurality of different types of vehicle sensors, determining latent features by extracting modality-specific features from the input data stream, and aligning a distribution of the latent features of different modalities by feature-level data fusion. Classification probabilities can be determined for the latent features using a fused modality scene classifier. A tree-organized neural network can be trained to determine path probabilities and issue driving pattern judgments, with the tree-organized neural network including a soft tree model and a hard decision leaf. One or more driving pattern judgments can be issued based on a probability of possible driving patterns derived from the modality-specific features.
1. A method for data fusion and analysis of vehicle sensor data, comprising:
receiving a multiple modality input data stream from a plurality of vehicle sensors;
determining one or more latent features by extracting one or more modality-specific features from the input data stream;
aligning a distribution of the latent features of different modalities by feature-level data fusion;
determining classification probabilities for one or more of the latent features vising a fused modality scene classifier;
training a tree-organized neural network to determine path probabilities and driving pattern judgments, the tree-organized neural network comprising a soft tree model and a hard decision leaf by measuring a similarity between latent features by an inner product after normalization;
issuing one or more driving pattern judgments based on a probability of possible driving patterns derived from the one or more modality-specific features to a pattern analyzer to determine a driving pattern output; and
controlling an operation of an autonomous vehicle based on the driving pattern output.
2. The method as recited in claim 1 , wherein the aligning the distribution of the latent features comprises performing adversarial regularization on the distribution of the latent features using a discriminator-based regularizer.
3. The method as recited in claim 1 , wherein the fused modality scene classifier comprises two fully connected layers and a softmax layer.
4. The method as recited in claim 1 , wherein the tree-organized neural network is configured for filtering patterns that are avoided in reality from the driving pattern judgments.
5. The method as recited in claim 1 , wherein each node in the tree-organized neural network is a neural network, and the neural network of the each node is optimized by back-propagation to minimize internal cross-entropy loss.
6. The method as recited in claim 1 , wherein the aligning the distribution of the latent features of different modalities comprises regularizing using one or more discriminator-based regularizers, with each of the regularizers comprising three fully connected layers.
7. A system for data fusion and analysis of vehicle sensor data, comprising:
one or more processors operatively coupled to a non-transitory computer-readable storage medium, the processors being configured for:
receiving a multiple modality input data stream from a plurality of vehicle sensors;
determining one or more latent features by extracting one or more modality-specific features from the input data stream;
aligning a distribution of the latent features of different modalities by feature-level data fusion;
determining classification probabilities for one or more of the latent features using a fused modality scene classifier;
training a tree-organized neural network to determine path probabilities and issue driving pattern judgments, the tree-organized neural network comprising a soft tree model and a hard decision leaf by measuring a similarity between latent features by an inner product after normalization;
issuing one or more driving pattern judgments based on a probability of possible driving patterns derived from the one or more modality-specific features to a pattern analyzer to determine a driving pattern output; and
controlling an operation of an autonomous vehicle based on the driving pattern output.
8. The system as recited in claim 7 , wherein the aligning the distribution of the latent features comprises performing adversarial regularization on the distribution of the latent features using a discriminator-based regularizer.
9. The system as recited in claim 7 , wherein the fused modality scene classifier comprises two fully connected layers and a softmax layer.
10. The system as recited in claim 7 , wherein the tree-organized neural network is configured for filtering patterns that are avoided in reality from the driving pattern judgments.
11. The system as recited in claim 7 , wherein each node in the tree-organized neural network is a neural network, and the neural network of the each node is optimized by back-propagation to minimize internal cross-entropy loss.
12. The system as recited in claim 7 , wherein the aligning the distribution of the latent features of different modalities comprises regularizing using one or more discriminator-based regularizers, with each of the regularizers comprising three fully connected layers.
13. A non-transitory computer readable storage medium comprising a computer readable program operatively coupled to a processor device for data fusion and analysis of vehicle sensor data, wherein the computer readable program when executed on a computer causes the computer to perform the steps of:
receiving a multiple modality input data stream from a plurality of vehicle sensors;
determining one or more latent features by extracting one or more modality-specific features from the input data stream;
aligning a distribution of the latent features of different modalities by feature-level data fusion;
determining classification probabilities for one or more of the latent features using a fused modality scene classifier;
training a tree-organized neural network to determine path probabilities and issue driving pattern judgments, the tree-organized neural network comprising a soft tree model and a hard decision leaf by measuring a similarity between latent features by an inner product after nominalization; and
issuing one or more driving pattern judgments based on a probability of possible driving patterns derived from the one or more modality-specific features to a pattern analyzer to determine a driving pattern output; and
controlling an operation of an autonomous vehicle based on the driving pattern output.
14. The non-transitory computer readable storage medium as recited in claim 13 , wherein the aligning the distribution of the latent features comprises performing adversarial regularization on the distribution of the latent features using a discriminator-based regularizer.
15. The non-transitory computer readable storage medium as recited in claim 11 , wherein the fused modality scene classifier comprises two fully connected layers and a softmax layer.
16. The non-transitory computer readable storage medium as recited in claim 13 , wherein the tree-organized neural network is configured for filtering patterns that are avoided in reality from the driving pattern judgments.
17. The non-transitory computer readable storage medium as recited in claim 13 , wherein the aligning the distribution of the latent features of different modalities comprises regularizing using one or more discriminator-based regularizers, with each of the regularizers comprising three fully connected layers.