Using simulations to identify differences between behaviors of manually-driven and autonomous vehicles

1. A method for adjusting software used to operate a simulated autonomous vehicle, the method comprising:

retrieving, by one or more processors, log data collected for a manually-driven vehicle driving along a route;

generating, by the one or more processors based on the retrieved log data, a plurality of path segments for a portion of the route, each of the plurality of path segments being defined by a position of the manually-driven vehicle at a start of a time segment and at an end of the time segment;

running, by the one or more processors using first software for autonomous driving, a simulation including the simulated autonomous vehicle that drives along the plurality of path segments to follow a same route as the manually-driven vehicle;

collecting, by the one or more processors, simulation data for the simulated autonomous vehicle;

extracting, by the one or more processors, one or more metrics from the log data collected for the manually-driven vehicle and one or more metrics collected from the simulation data collected for the simulated autonomous vehicle;

computing, by the one or more processors, a first distance traveled by the manually-driven vehicle along the portion of the route within an amount of time based on the one or more metrics extracted from the log data;

computing, by the one or more processors, a second distance traveled by the simulated autonomous vehicle along the portion of the route within the amount of time based on the one or more metrics extracted from the simulated data;

determining, by the one or more processors, a difference between the first distance and the second distance; and

in response to determining the difference, adjusting, by the one or more processors, the first software to operate the simulated autonomous vehicle more similarly to the manually-driven vehicle.

2. The method of claim 1 , wherein the one or more metrics include at least one of a distance metric, a time metric or a speed metric.

3. The method of claim 1 , wherein the adjusting the first software includes altering a passenger discomfort level permitted for a type of location or a type of maneuver associated with the portion of the route.

4. The method of claim 1 , wherein the adjusting the first software includes altering an amount of buffer given to a type of object.

5. The method of claim 1 , further comprising:

running, by the one or more processors, the simulation using a second software;

determining, for the second software, by the one or more processors, a second difference between the first distance traveled by the manually-driven vehicle along the portion of the route within the amount of time and a third distance traveled by a second simulated autonomous vehicle along the portion of the route within the amount of time; and

in response to determining the second difference, selecting, by the one or more processors, one of the first software or the second software based on which of the second distance traveled by the simulated autonomous vehicle or the third distance traveled by the second simulated autonomous vehicle is closest to the first distance traveled by the manually-driven vehicle.

6. The method of claim 1 , further comprising estimating, by the one or more processors, a frequency of progress issues for the simulated autonomous vehicle.

7. The method of claim 1 , further comprising selecting, by the one or more processors, the portion of the route based on a type of location.

8. The method of claim 1 , further comprising selecting, by the one or more processors, the portion of the route based on user input.

9. A non-transitory, tangible computer-readable medium on which computer-readable instructions of a program are stored, the instructions, when executed by one or more computing devices, cause the one or more computing devices to perform a method for adjusting software used to operate a simulated autonomous vehicle, the method comprising:

retrieving log data collected for a manually-driven vehicle driving along a route;

generating, based on the retrieved log data, a plurality of path segments for a portion of the route, each of the plurality of path segments being defined by a position of the manually-driven vehicle at a start of a time segment and at an end of the time segment;

running, using first software for autonomous driving, a simulation including the simulated autonomous vehicle that drives along the plurality of path segments to follow a same route as the manually-driven vehicle;

collecting simulation data for the simulated autonomous vehicle;

extracting one or more metrics from the log data collected for the manually-driven vehicle and one or more metrics collected from the simulation data collected for the simulated autonomous vehicle;

computing a first distance traveled by the manually-driven vehicle along the portion of the route within an amount of time based on the one or more metrics extracted from the log data;

computing a second distance traveled by the simulated autonomous vehicle along the portion of the route within the amount of time based on the one or more metrics extracted from the simulated data;

determining a difference between the first distance and the second distance; and

in response to determining the difference, adjusting the first software to operate the simulated autonomous vehicle more similarly to the manually-driven vehicle.

10. The medium of claim 9 , wherein the one or more metrics include at least one of a distance metric, a time metric or a speed metric.

11. The medium of claim 9 , wherein the adjusting the first software includes altering a passenger discomfort level permitted for a type of location or a type of maneuver associated with the portion of the route.

12. The medium of claim 9 , wherein the adjusting the first software includes altering an amount of buffer given to a type of object.

13. The medium of claim 9 , further comprising:

running the simulation using a second software;

determining, for the second software, a second difference between the first distance traveled by the manually-driven vehicle along the portion of the route within the amount of time and a third distance traveled by a second simulated autonomous vehicle along the portion of the route within the amount of time; and

in response to determining the second difference, selecting one of the first software or the second software based on which of the second distance traveled by the simulated autonomous vehicle or the third distance traveled by the second simulated autonomous vehicle that is closest to the distance traveled by the manually-driven vehicle.

14. The medium of claim 9 , wherein the method further comprises estimating a frequency of progress issues for the simulated autonomous vehicle.

15. The medium of claim 9 , wherein the method further comprises selecting the portion of the route based on a type of location.

16. The medium of claim 9 , wherein the method further comprises selecting the portion of the route based on user input.

17. The method of claim 1 , wherein a length of time between the start of the time segment and the end of the time segment is same for each of the plurality of path segments.

18. The medium of claim 9 , wherein a length of time between the start of the time segment and the end of the time segment is same for each of the plurality of path segments.