Systems and methods for aligning trajectory information derived from different source data

1. A computer-implemented method comprising:

obtaining sensor data captured by a vehicle during a given period of operation, wherein the captured sensor data comprises (i) a first set of sensor data captured by a first sensor system associated with the vehicle, and (ii) a second set of sensor data captured by a second sensor system associated with the vehicle, wherein the first sensor system is of a first type that is being evaluated for use in collecting trajectories and the second sensor system is of a second type that has previously been validated for use in collecting trajectories;

deriving (i) a first representation of an agent's trajectory from the first set of sensor data captured by the first sensor system associated with the vehicle and (ii) a second representation of the agent's trajectory from the second set of sensor data captured by the second sensor system associated with the vehicle, wherein the first and second representations of the agent's trajectory are based on different spatial reference frames and different temporal reference frames;

aligning the spatial reference frames of the first and second representations by applying a spatial transformation to a given one of the first and second representations;

aligning the temporal reference frames of the first and second representations by determining an origin-time offset between the temporal reference frames of the first and second representations and applying the determined origin-time offset to timing information encoded in the given one of the first and second representations;

after aligning the spatial reference frames and the temporal references frames of the first and second representations, performing a comparison between the aligned first and second representations; and

using the comparison between the aligned first and second representations as a basis for evaluating an accuracy of trajectories collected using the first type of sensor system relative to trajectories collected using the second type of sensor system that has previously been validated for use in collecting trajectories.

2. The computer-implemented method of claim 1 , wherein aligning the spatial reference frames of the first and second representations by applying the spatial transformation to the given one of the first and second representations comprises:

identifying a spatial transformation to the given one of the first and second representations that achieves a best geometric match between the first representation and the second representation; and

applying the identified spatial transformation to the given one of the first and second representations.

3. The computer-implemented method of claim 1 , wherein determining the origin-time offset between the temporal reference frames of the first and second representations comprises:

identifying the origin-time offset that minimizes a positional error between a set of position and orientation (“pose”) values included in the first representation and a counterpart set of pose values included in the second representation.

4. The computer-implemented method of claim 1 , wherein aligning the temporal reference frames of the first and second representations further comprises:

aligning a first time-interval scale of the first representation with a second time-interval scale of the second representation.

5. The computer-implemented method of claim 4 , wherein aligning the first time-interval scale of the first representation with the second time-interval scale of the second representation comprises:

adding interpolated pose values to at least one of the first representation or the second representation.

6. The computer-implemented method of claim 1 , wherein aligning the spatial reference frames of the first and second representations further comprises:

identifying a rotation amount that minimizes an orientational error between a set of position and orientation (“pose”) values included in the first representation and a counterpart set of pose values included in the second representation; and

applying the identified rotation amount to each pose value of the set of pose values included in the given one of the first and second representations.

7. The computer-implemented method of claim 1 , wherein performing the comparison between the aligned first and second representations comprises:

determining an error metric that indicates an extent to which the first representation differs from the second representation.

8. The computer-implemented method of claim 1 , further comprising, after aligning the spatial reference frames and temporal reference frames of the first and second representations:

using the first representation of the agent's trajectory to derive a first dataset characterizing a given instance of a scenario type;

using the second representation of the agent's trajectory to derive a second dataset characterizing the given instance of the scenario type;

including the first and second datasets characterizing the given instance of the scenario type as part of a comparison between data characterizing instances of the scenario type derived from sensor data captured by the first type of sensor system and data characterizing instances of the scenario type derived from sensor data captured by the second type of sensor system;

based on the comparison, determining an error between the data characterizing instances of the scenario type derived from the first set of sensor data captured by the first type of sensor system and the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system; and

using the determined error to evaluate an accuracy level of data characterizing instances of the scenario type derived from the sensor data captured by the first type of sensor system relative to the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system.

9. A non-transitory computer-readable medium comprising program instructions stored thereon that, when executed by at least one processor of a computing system, cause the computing system to perform functions comprising:

obtaining sensor data captured by a vehicle during a given period of operation, wherein the captured sensor data comprises (i) a first set of sensor data captured by a first sensor system associated with the vehicle, and (ii) a second set of sensor data captured by a second sensor system associated with the vehicle, wherein the first sensor system is of a first type that is being evaluated for use in collecting trajectories and the second sensor system is of a second type that has previously been validated for use in collecting trajectories;

deriving (i) a first representation of an agent's trajectory from the first set of sensor data captured by the first sensor system associated with the vehicle and (ii) a second representation of the agent's trajectory from the second set of sensor data captured by the second sensor system associated with the vehicle, wherein the first and second representations of the agent's trajectory are based on different spatial reference frames and different temporal reference frames;

aligning the spatial reference frames of the first and second representations by applying a spatial transformation to a given one of the first and second representations;

aligning the temporal reference frames of the first and second representations by determining an origin-time offset between the temporal reference frames of the first and second representations and applying the determined origin-time offset to timing information encoded in the given one of the first and second representations;

after aligning the spatial reference frames and the temporal references frames of the first and second representations, performing a comparison between the aligned first and second representations; and

using the comparison between the aligned first and second representations as a basis for evaluating an accuracy of trajectories collected using the first type of sensor system relative to trajectories collected using the second type of sensor system that has previously been validated for use in collecting trajectories.

10. The non-transitory computer-readable medium of claim 9 , wherein aligning the spatial reference frames of the first and second representations by applying the spatial transformation to the given one of the first and second representations comprises:

identifying a spatial transformation to the given one of the first and second representations that achieves a best geometric match between the first representation and the second representation; and

applying the identified spatial transformation to the given one of the first and second representations.

11. The non-transitory computer-readable medium of claim 9 , wherein determining the origin-time offset between the temporal reference frames of the first and second representations comprises:

identifying the origin-time offset that minimizes a positional error between a set of position and orientation (“pose”) values included in the first representation and a counterpart set of pose values included in the second representation.

12. The non-transitory computer-readable medium of claim 9 , wherein aligning the temporal reference frames of the first and second representations further comprises:

aligning a first time-interval scale of the first representation with a second time-interval scale of the second representation.

13. The non-transitory computer-readable medium of claim 12 , wherein aligning the first time-interval scale of the first representation with the second time-interval scale of the second representation comprises:

adding interpolated pose values to at least one of the first representation or the second representation.

14. The non-transitory computer-readable medium of claim 9 , wherein aligning the spatial reference frames of the first and second representations further comprises:

identifying a rotation amount that minimizes an orientational error between a set of position and orientation (“pose”) values included in the first representation and a counterpart set of pose values included in the second representation; and

applying the identified rotation amount to each pose value of the set of pose values included in the given one of the first and second representations.

15. The non-transitory computer-readable medium of claim 9 , wherein performing the comparison between the aligned first and second representations comprises:

determining an error metric that indicates an extent to which the first representation differs from the second representation.

16. The non-transitory computer-readable medium of claim 9 , further comprising program instructions stored thereon that, when executed by the at least one processor of the computing system, cause the computing system to perform functions comprising, after aligning the spatial reference frames and temporal reference frames of the first and second representations:

using the first representation of the agent's trajectory to derive a first dataset characterizing a given instance of a scenario type;

using the second representation of the agent's trajectory to derive a second dataset characterizing the given instance of the scenario type;

including the first and second datasets characterizing the given instance of the scenario type as part of a comparison between data characterizing instances of the scenario type derived from sensor data captured by the first type of sensor system and data characterizing instances of the scenario type derived from sensor data captured by the second type of sensor system;

based on the comparison, determining an error between the data characterizing instances of the scenario type derived from the sensor data captured by the first type of sensor system and the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system; and

using the determined error to evaluate an accuracy level of data characterizing instances of the scenario type derived from the sensor data captured by the first type of sensor system relative to the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system.

17. A computing system comprising:

at least one processor;

a non-transitory computer-readable medium; and

program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is configured to:

obtain sensor data captured by a vehicle during a given period of operation, wherein the captured sensor data comprises (i) a first set of sensor data captured by a first sensor system associated with the vehicle, and (ii) a second set of sensor data captured by a second sensor system associated with the vehicle, wherein the first sensor system is of a first type that is being evaluated for use in collecting trajectories and the second sensor system is of a second type that has previously been validated for use in collecting trajectories;

derive (i) a first representation of an agent's trajectory from the first set of sensor data captured by the first sensor system associated with the vehicle and (ii) a second representation of the agent's trajectory from the second set of sensor data captured by the second sensor system associated with the vehicle, wherein the first and second representations of the agent's trajectory are based on different spatial reference frames and different temporal reference frames;

align the spatial reference frames of the first and second representations by applying a spatial transformation to a given one of the first and second representations;

align the temporal reference frames of the first and second representations by determining an origin-time offset between the temporal reference frames of the first and second representations and applying the determined origin-time offset to timing information encoded in the given one of the first and second representations;

after aligning the spatial reference frames and the temporal references frames of the first and second representations, perform a comparison between the aligned first and second representations; and

use the comparison between the aligned first and second representations as a basis for evaluating an accuracy of trajectories collected using the first type of sensor system relative to trajectories collected using the second type of sensor system that has previously been validated for use in collecting trajectories.

18. The computing system of claim 17 , further comprising program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is configured to:

use the first representation of the agent's trajectory to derive a first dataset characterizing a given instance of a scenario type;

use the second representation of the agent's trajectory to derive a second dataset characterizing the given instance of the scenario type;

include the first and second datasets characterizing the given instance of the scenario type as part of a comparison between data characterizing instances of the scenario type derived from sensor data captured by the first type of sensor system and data characterizing instances of the scenario type derived from sensor data captured by the second type of sensor system;

based on the comparison, determine an error between the data characterizing instances of the scenario type derived from the sensor data captured by the first type of sensor system and the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system; and

use the determined error to evaluate an accuracy level of data characterizing instances of the scenario type derived from the sensor data captured by the first type of sensor system relative to the data characterizing instances of the scenario type derived from the sensor data captured by the second type of sensor system.

19. The computing system of claim 17 , wherein the program instructions that are executable by the at least one processor such that the computing system is configured to align the spatial reference frames of the first and second representations by applying the spatial transformation to the given one of the first and second representations comprise program instructions that are executable by the at least one processor such that the computing system is configured to:

identify a spatial transformation to the given one of the first and second representations that achieves a best geometric match between the first representation and the second representation; and

apply the identified spatial transformation to the given one of the first and second representations.

20. The computing system of claim 17 , wherein the program instructions that are executable by the at least one processor such that the computing system is configured to determine the origin-time offset between the temporal reference frames of the first and second representations comprise program instructions that are executable by the at least one processor such that the computing system is configured to:

identify the origin-time offset that minimizes a positional error between a set of position and orientation (“pose”) values included in the first representation and a counterpart set of pose values included in the second representation.