Fusion of Object Classes
This document describes techniques and systems for fusion of object classes. Sensor data, related to an object, can be obtained from different types of sensor systems. Using techniques based on a Dempster Shafer framework, masses of each potential object class based on data from a respective sensor can be calculated. The respective masses of each potential object class, based on the different sensor systems, can be combined to generate a fused mass for each potential object class. Probabilities for each potential object class can be calculated based on the respective fused masses. An assigned object class for the object can be selected based on a probability exceeding some decision threshold. This assigned object class may then be available for an object tracker, an object fusion system, or other vehicle system resulting in a safer driving experience.
1 . A method comprising:
obtaining first sensor data from a first sensor and second sensor data from a second sensor that is a different type of sensor than the first sensor, the first sensor and the second sensor having at least partially overlapping fields of view;
determining, for an object in the at least partially overlapping fields of view, a set of first masses based on the first sensor data, each first mass in the set of first masses being associated with assigning a respective object class from a set of potential object classes to the object;
determining, for the object in the at least partially overlapping fields of view, a set of second masses based on the second sensor data, each second mass in the set of second masses being associated with assigning a respective object class from the set of potential object classes to the object;
combining the respective first mass and the respective second mass of the respective object class from the set of potential object classes to generate a fused mass for the respective object class;
determining, based on the fused mass of each object class in the set of potential object classes, a probability for each object class being an assigned object class for the object;
selecting, based on a respective probability of an object class exceeding a decision threshold, the respective object class associated with the respective probability to be the assigned object class of the object; and
outputting, to a semi-autonomous or autonomous driving system of a vehicle, the assigned object class of the object to control operation of the vehicle.
2 . The method of claim 1 , wherein the first sensor data and the second sensor data comprise:
an initial object class assigned to the object; and
contextual information that is categorized as evidence factors and trust factors.
3 . The method of claim 2 , wherein:
the evidence factors comprise motion cues related to the object and size cues related to the object; and
the trust factors comprise:
a stability value of the initial object class;
an existence probability related to an existence of the object;
a failsafe probability related to a performance of a respective sensor; and
a field of view probability related to an accuracy of the initial object class based on where in a field of view of the respective sensor the object is located.
4 . The method of claim 3 , wherein determining the set of first masses and the set of second masses comprises:
determining a first probability of the initial object class obtained from the respective sensor data;
determining a second probability based on the evidence factors of the respective sensor data;
determining a third probability based on the trust factors of the respective sensor data; and
combining the first probability, the second probability, and the third probability to generate the set of masses for the respective sensor.
5 . The method of claim 4 , wherein determining the second probability comprises:
determining, for each class in the set of potential classes, a motion probability based on the motion cues;
determining, for each class in the set of potential classes, a size probability based on the size cues; and
combining the motion probability and the size probability.
6 . The method of claim 4 , wherein determining the third probability comprises:
determining, for each class in the set of potential classes, the existence probability;
determining, for each class in the set of potential classes, the failsafe probability;
determining, for each class in the set of potential classes, the field of view probability; and
combining the existence probability, the failsafe probability, and the field of view probability.
7 . The method of claim 1 , wherein Dempster's combination rule is used to combine the respective first mass and the respective second mass of the object class.
8 . The method of claim 1 , wherein determining the probability for each object class being the assigned object class for the object comprises:
calculating, based on the fused mass of each object class in the set of potential object classes, a belief parameter and a plausibility parameter related to the object class; and
averaging the belief parameter and the plausibility parameter.
9 . The method of claim 8 , wherein calculating the belief parameter for a particular object class comprises:
determining all subsets in the set of potential object classes that include the particular object class; and
summing the fused masses of all the subsets that include the particular object class.
10 . The method of claim 8 , wherein calculating the plausibility parameter for a particular object class comprises:
determining non-zero intersections of the particular object class with other object classes; and
summing the fused masses of all object classes having the non-zero intersections.
11 . The method of claim 8 , further comprising:
implementing a low-pass filter on the average of the belief parameter and the plausibility parameter to stabilize a respective object class temporally.
12 . The method of claim 1 , wherein the set of potential object classes comprises:
car;
truck;
motorcycle;
bicycle; and
pedestrian.
13 . The method claim 12 , wherein the set of potential object classes further comprise potential power classes, the potential power classes including:
car or truck;
motorcycle or bicycle;
bicycle or pedestrian;
motorcycle or pedestrian; and
undetermined.
14 . A system comprising:
one or more processors configured to:
obtain first sensor data from a first sensor and second sensor data from a second sensor that is a different type of sensor than the first sensor, the first sensor and the second sensor having at least partially overlapping fields of view;
determine, for an object in the at least partially overlapping fields of view, a set of first masses based on the first sensor data, each first mass in the set of first masses being associated with assigning a respective object class from a set of potential object classes to the object;
determine, for the object in the at least partially overlapping fields of view, a set of second masses based on the second sensor data, each second mass in the set of second masses being associated with assigning a respective object class from the set of potential object classes to the object;
combine the respective first mass and the respective second mass of the respective object class from the set of potential object classes to generate a fused mass for the respective object class;
determine, based on the fused mass of each object class in the set of potential object classes, a probability for each object class being an assigned object class for the object;
select, based on a respective probability of an object class exceeding a decision threshold, the respective object class associated with the respective probability to be the assigned object class of the object; and
output, to a semi-autonomous or autonomous driving system of a vehicle, the assigned object class of the object to control operation of the vehicle.
15 . The system of claim 14 , wherein the first sensor data and the second sensor data comprise:
an initial object class assigned to the object; and
contextual information that is categorized as evidence factors and trust factors.
16 . The system of claim 15 , wherein:
the evidence factors comprise motion cues related to the object and size cues related to the object; and
the trust factors comprise:
a stability value of the initial object class;
an existence probability related to an existence of the object;
a failsafe probability related to a performance of a respective sensor; and
a field of view probability related to an accuracy of the initial object class based on where in a field of view of the respective sensor the object is located.
17 . The system of claim 16 , wherein the one or more processors are configured to determine the set of first masses and the set of second masses by at least:
determining a first probability of the initial object class obtained from the respective sensor data;
determining a second probability based on the evidence factors of the respective sensor data;
determining a third probability based on the trust factors of the respective sensor data; and
combining the first probability, the second probability, and the third probability to generate the set of masses for the respective sensor.
18 . The system of claim 17 , wherein determining the second probability comprises:
determining, for each class in the set of potential classes, a motion probability based on the motion cues;
determining, for each class in the set of potential classes, a size probability based on the size cues; and
combining the motion probability and the size probability.
19 . The system of claim 17 , wherein determining the third probability comprises:
determining, for each class in the set of potential classes, the existence probability;
determining, for each class in the set of potential classes, the failsafe probability;
determining, for each class in the set of potential classes, the field of view probability; and
combining the existence probability, the failsafe probability, and the field of view probability.
20 . A computer-readable media comprising instructions that, when executed, cause a processor to:
obtain first sensor data from a first sensor and second sensor data from a second sensor that is a different type of sensor than the first sensor, the first sensor and the second sensor having at least partially overlapping fields of view;
determine, for an object in the at least partially overlapping fields of view, a set of first masses based on the first sensor data, each first mass in the set of first masses being associated with assigning a respective object class from a set of potential object classes to the object;
determine, for the object in the at least partially overlapping fields of view, a set of second masses based on the second sensor data, each second mass in the set of second masses being associated with assigning a respective object class from the set of potential object classes to the object;
combine the respective first mass and the respective second mass of the respective object class from the set of potential object classes to generate a fused mass for the respective object class;
determine, based on the fused mass of each object class in the set of potential object classes, a probability for each object class being an assigned object class for the object;
select, based on a respective probability of a respective object class exceeding a decision threshold, the respective object class associated with the respective probability to be the assigned object class of the object; and
output, to a semi-autonomous or autonomous driving system of a vehicle, the assigned object class of the object to control operation of the vehicle.