Method and system for modeling personalized car-following driving styles with model-free inverse reinforcement learning
A method may include learning reward functions for a plurality of first vehicles based on first vehicle data associated with the plurality of first vehicles using inverse reinforcement learning, associating each vehicle of the plurality of first vehicles with one cluster among a plurality of clusters based on the reward functions, determining a centroid reward function for each of the clusters based on the reward functions associated with each cluster, performing a comparison between second vehicle data associated with a second vehicle and the first vehicle data, determining a vehicle among the plurality of first vehicles having associated first vehicle data that is most similar to the second vehicle data based on the comparison, associating the second vehicle with the cluster associated with the determined vehicle, and controlling operation of the second vehicle based on the centroid reward function of the cluster associated with the second vehicle.
1 . A method comprising:
learning reward functions for a plurality of first vehicles based on first vehicle data associated with the plurality of first vehicles using inverse reinforcement learning;
performing cluster analysis to associate each vehicle of the plurality of first vehicles with one cluster of reward functions among a plurality of clusters of reward functions based on the learned reward functions;
determining a centroid reward function for each of the clusters of reward functions based on the reward functions associated with each cluster of reward functions;
performing a comparison between second vehicle data associated with a second vehicle and the first vehicle data;
determining a vehicle among the plurality of first vehicles that is most similar to the second vehicle data based on the comparison;
associating the second vehicle with the cluster of reward functions associated with the determined vehicle; and
controlling operation of the second vehicle based on the centroid reward function of the cluster of reward functions associated with the second vehicle.
2 . The method of claim 1 , further comprising extracting a vehicle event as a vehicle following event and obtaining the first vehicle data associated with a first vehicle when the first vehicle has a speed within a threshold range and a distance between the first vehicle and a leading vehicle driving ahead of the first vehicle is less than a threshold distance.
3 . The method of claim 1 , wherein each of the reward functions comprises a weight for each of a plurality of states associated with corresponding first vehicle.
4 . The method of claim 3 , wherein each state of the plurality of states comprises corresponding first vehicle having a speed within a particular speed range and a following distance behind a leading vehicle within a particular distance range.
5 . The method of claim 1 , further comprising associating each vehicle of the plurality of first vehicles with one cluster among the plurality of clusters using K-means clustering.
6 . The method of claim 1 , further comprising determining the centroid reward function for each of the clusters by averaging weights of each of the reward functions associated with each cluster.
7 . The method of claim 1 , further comprising performing the comparison between the second vehicle data associated with the second vehicle and the first vehicle data by:
determining an output of a Gaussian mixture model for each vehicle of the plurality of first vehicles based on the first vehicle data;
determining a second output of the Gaussian mixture model for the second vehicle based on the second vehicle data; and
comparing the second output of the Gaussian mixture model for the second vehicle to the output of the Gaussian mixture model for each vehicle of the plurality of first vehicles.
8 . The method of claim 7 , further comprising:
comparing the second output of the Gaussian mixture model for the second vehicle to the output of the Gaussian mixture model for each vehicle of the plurality of first vehicles based on a Kullback-Leibler (KL)-divergence.
9 . The method of claim 8 , further comprising using Monte-Carlo sampling to approximate the KL-divergence.
10 . The method of claim 1 , further comprising:
determining inertia and distortion based on the number of clusters; and
determining the number of clusters based on a plot of the inertia and the distortion.
11 . The method of claim 1 , further comprising:
receiving third vehicle data associated with the second vehicle driving autonomously;
determining a driving action to perform based on the third vehicle data and the centroid reward function of the cluster associated with the second vehicle; and
causing the second vehicle to perform the driving action.
12 . The method of claim 11 , wherein the third vehicle data comprises a speed of the second vehicle, a speed of a leading vehicle driving in front of the second vehicle, and a distance between the second vehicle and the leading vehicle.
13 . The method of claim 11 , wherein the driving action comprises an amount to accelerate or decelerate the second vehicle.
14 . The method of claim 11 , further comprising using a cost-constrained partially observable Monte-Carlo planner to determine the driving action.
15 . A computing device comprising a controller programmed to:
learn reward functions for a plurality of first vehicles based on first vehicle data associated with the plurality of first vehicles using inverse reinforcement learning;
perform cluster analysis to associate each vehicle of the plurality of first vehicles with one cluster of reward functions among a plurality of clusters of reward functions based on the learned reward functions;
determine a centroid reward function for each of the clusters of reward functions based on the reward functions associated with each cluster of reward functions;
perform a comparison between second vehicle data associated with a second vehicle and the first vehicle data;
determine a vehicle among the plurality of first vehicles that is most similar to the second vehicle data based on the comparison;
associate the second vehicle with the cluster of reward functions associated with the determined vehicle; and
control operation of the second vehicle based on the centroid reward function of the cluster of reward functions associated with the second vehicle.
16 . The computing device of claim 15 , wherein the controller is further programmed to determine the centroid reward function for each of the clusters by averaging weights of each of the reward functions associated with each cluster.
17 . The computing device of claim 15 , wherein the controller is further programmed to perform the comparison between the second vehicle data associated with the second vehicle and the first vehicle data by:
determining an output of a Gaussian mixture model for each vehicle of the plurality of first vehicles based on the first vehicle data;
determining a second output of the Gaussian mixture model for the second vehicle based on the second vehicle data; and
comparing the second output of the Gaussian mixture model for the second vehicle to the output of the Gaussian mixture model for each vehicle of the plurality of first vehicles.
18 . The computing device of claim 17 , wherein the controller is further programmed to:
compare the second output of the Gaussian mixture model for the second vehicle to the output of the Gaussian mixture model for each vehicle of the plurality of first vehicles based on a KL-divergence.
19 . The computing device of claim 15 , wherein the controller is further programmed to:
receive third vehicle data associated with the second vehicle driving autonomously;
determine a driving action to perform based on the third vehicle data and the centroid reward function of the cluster associated with the second vehicle; and
cause the second vehicle to perform the driving action.
20 . The computing device of claim 19 , wherein the computing device is further programmed to use a cost-constrained partially observable Monte-Carlo planner to determine the driving action.