Systems and methods for evaluating object tracking methodologies

1 . An autonomous vehicle comprising:

one or more sensors;

at least one processor; and

at least one memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

receiving sensor data from the one or more sensors of the autonomous vehicle;

determining, using a first tracking methodology, methodology tracking based on the sensor data;

receiving ground truth tracking;

determining an association between the methodology tracking and the ground truth tracking, determining the association further comprising generating an association table that counts a number of times that the methodology tracking is associated with the ground truth tracking;

determining a joint probability distribution of the methodology tracking and the ground truth tracking;

determining a methodology tracking distribution based on the joint probability distribution;

determining a ground truth tracking distribution based on the joint probability distribution;

determining a Kullback-Leibler Divergence (KLD) score between the methodology tracking distribution and the ground truth tracking distribution; and

controlling operation of the autonomous vehicle based on the KLD score.

2 . The autonomous vehicle of claim 1 , wherein the methodology tracking includes the first tracking methodology and a second tracking methodology and wherein the KLD score includes a first KLD score and a second KLD score, and wherein the at least one memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

determining the first KLD score for the first tracking methodology;

determining the second KLD score for the second tracking methodology;

comparing the first KLD score and the second KLD score; and

based on the comparison, selecting between the first tracking methodology and the second tracking methodology implemented on the autonomous vehicle.

3 . The autonomous vehicle of claim 1 , wherein the at least one memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

optimizing the methodology by optimizing the KLD score.

4 . The autonomous vehicle of claim 1 , wherein the at least one memory storing instructions, which, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

determining the association by:

computing a similarity metric between the methodology tracking and the ground truth tracking; and

comparing the similarity metric with a threshold.

5 . The autonomous vehicle of claim 1 , wherein determining the methodology tracking distribution comprises integrating the joint probability distribution over the ground truth tracking.

6 . The autonomous vehicle of claim 1 , wherein determining the ground truth tracking distribution comprises integrating the joint probability distribution over the methodology tracking.

7 . A computer implemented method comprising:

receiving sensor data from one or more sensors of an autonomous vehicle;

determining, using a first tracking methodology, methodology tracking based on the sensor data;

receiving ground truth tracking;

determining an association between the methodology tracking and the ground truth tracking;

determining a joint probability distribution of the methodology tracking and the ground truth tracking;

determining a methodology tracking distribution based on the joint probability distribution;

determining a ground truth tracking distribution based on the joint probability distribution, determining the ground truth tracking distribution further comprising integrating the joint probability distribution over the methodology tracking;

determining a Kullback-Leibler Divergence (KLD) score between the methodology tracking distribution and the ground truth tracking distribution; and

controlling operation of the autonomous vehicle based on the KLD score.

8 . The computer implemented method of claim 7 , wherein the methodology tracking includes the first tracking methodology and a second tracking methodology and wherein the KLD score includes a first KLD score and a second KLD score, and wherein the method further comprises:

determining the first KLD score for the first tracking methodology;

determining the second KLD score for the second tracking methodology, and comparing the first KLD score and the second KLD score; and

based on the comparison, selecting between the first tracking methodology and the second tracking methodology implemented on the autonomous vehicle.

9 . The computer implemented method of claim 7 , wherein controlling the operation further comprises:

optimizing the methodology by optimizing the KLD score.

10 . The computer implemented method of claim 7 , wherein determining the association further comprises:

computing a similarity metric between the methodology tracking and the ground truth tracking; and

determining the association by comparing the similarity metric with a threshold.

11 . The computer implemented method of claim 7 , determining the association comprises:

generating an association table that counts a number of times that the methodology tracking matches the ground truth tracking, for each parameter, for each time step, for each actor.

12 . The computer implemented method of claim 7 , wherein determining the methodology tracking distribution comprises integrating the joint probability distribution over the ground truth tracking.

13 . A non-transitory computer-readable medium embodying programmed instructions which, when executed by at least one processor of an autonomous vehicle, cause the at least one processor to perform operations comprising:

receiving sensor data from one or more sensors of the autonomous vehicle;

determining, using a first tracking methodology, methodology tracking based on the sensor data;

receiving ground truth tracking;

determining an association between the methodology tracking and the ground truth tracking;

determining a joint probability distribution of the methodology tracking and the ground truth tracking;

determining a methodology tracking distribution based on the joint probability distribution;

determining a ground truth tracking distribution based on the joint probability distribution;

determining a Kullback-Leibler Divergence (KLD) score between the methodology tracking distribution and the ground truth tracking distribution;

controlling operation of the autonomous vehicle based on the KLD score;

determining a first KLD score for the first tracking methodology;

determining a second KLD score for a second tracking methodology;

comparing the first KLD score and the second KLD score; and

based on the comparison, selecting between the first tracking methodology and the second tracking methodology implemented on the autonomous vehicle.

14 . The non-transitory computer-readable medium of claim 13 , wherein the programmed instructions cause the at least one processor to perform operations comprising:

optimizing the methodology by optimizing the KLD score.

15 . The non-transitory computer-readable medium of claim 13 , wherein determining the association further comprises:

computing a similarity metric between the methodology tracking and the ground truth tracking; and

determining the association by comparing the similarity metric with a threshold.

16 . The non-transitory computer-readable medium of claim 13 , determining the association comprises:

generating an association table that counts a number of times that the methodology tracking matches the ground truth tracking, for each parameter, for each time step, for each actor.

17 . The non-transitory computer-readable medium of claim 13 , wherein determining the methodology tracking distribution comprises integrating the joint probability distribution over the ground truth tracking.

18 . The non-transitory computer-readable medium of claim 13 , wherein:

determining the ground truth tracking distribution further comprises integrating the joint probability distribution over the methodology tracking.