Object detection using similarity determinations for sensor observations
Techniques for performing object detection in a vehicle environment using sensor data captured by one or more sensors of the vehicle are described herein. In some cases, an object in a vehicle environment can be detected based on at least one of (i) a first similarity matrix that represents a first similarity value for two sensor observations associated with the vehicle environment, or (ii) a second similarity matrix that represents a second similarity value for a sensor observation and a track associated with the vehicle environment.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
receiving a first sensor observation based on first sensor data representing an environment of an autonomous vehicle;
receiving a second sensor observation based on second sensor data representing the environment, wherein the first sensor observation and the second sensor observation represent an object in the environment;
determining a first similarity matrix based on the first sensor observation, the second sensor observation, and a machine learning model, wherein the first similarity matrix includes a first similarity value, and wherein the first similarity value represents a first similarity of the first sensor observation and the second sensor observation;
determining, based on the first similarity matrix, a track associated with the object in the environment;
determining a second similarity matrix that includes a second similarity value, wherein the second similarity value represents a second similarity of a third sensor observation and the track; and
controlling the autonomous vehicle based on the second similarity matrix.
2. The system of claim 1 , wherein the first similarity value is determined based on a distance between a first embedding of the first sensor observation and a second embedding of the second sensor observation in an embedding space.
3. The system of claim 1 , wherein the first similarity matrix is associated with a set of sensor observations comprising the first sensor observation and the second sensor observation, and determining the track comprises:
determining a set of initial of tracks in the environment; and
assigning the set of sensor observations to the set of initial tracks based on the first similarity matrix.
4. The system of claim 1 , wherein the second similarity matrix is associated with a set of tracks comprising the track and a set of sensor observations comprising the third sensor observation and a fourth sensor observation, and the operations further comprise:
determining, based on the second similarity matrix represents, that the fourth sensor observation is not assigned to any of the set of tracks.
5. The system of claim 1 , wherein:
the first similarity matrix represents first similarity scores between individual observations of a plurality of sensor observations, and
the second similarity matrix represents second similarity scores between the individual observations and individual tracks.
6. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving a first sensor observation based on first sensor data representing an environment of a vehicle;
receiving a second sensor observation based on second sensor data representing the environment, wherein the first sensor observation and the second sensor observation represent an object in the environment;
determining, based on a machine learning model, a first similarity matrix including a first similarity value, wherein the first similarity value represents a first similarity of the first sensor observation and the second sensor observation;
determining a track associated with the environment based on the first similarity matrix, wherein the track represents the object;
determining a second similarity matrix that includes a second similarity value, wherein the second similarity value represents a second similarity of a third sensor observation and the track; and
controlling the vehicle based on the second similarity matrix.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the first similarity value is determined based on a distance between a first embedding of the first sensor observation and a second embedding of the second sensor observation in an embedding space.
8. The one or more non-transitory computer-readable media of claim 6 , wherein determining the track based on the first similarity matrix comprises:
determining that the track is associated with the environment at a previous time.
9. The one or more non-transitory computer-readable media of claim 8 , wherein the first similarity matrix is associated with a set of sensor observations comprising the first sensor observation and the second sensor observation, and wherein determining that the track is associated with the environment at the previous time comprises:
determining a set of initial tracks in the environment; and
assigning the set of sensor observations to the set of initial tracks based on the first similarity matrix.
10. The one or more non-transitory computer-readable media of claim 6 , wherein the first sensor observation and the second sensor observation are determined to not be associated with any individual tracks.
11. The one or more non-transitory computer-readable media of claim 10 , wherein determining that the first sensor observation is not associated with any individual tracks comprises:
determining that the second similarity matrix represents, for the first sensor observation and a second track, a third similarity value, wherein the second track is determined based on sensor observation data associated with a previous time; and
determining, based on the third similarity value, that the first sensor observation is not associated with any individual tracks.
12. The one or more non-transitory computer-readable media of claim 6 , wherein the first sensor observation comprises at least one of:
a lidar-based observation determined based on lidar data,
a radar-based observation determined based on radar data,
a sonar-based observation determined based on sonar data,
an image-based observation determined based on image data, or
one or more observations determined by detecting correlations across the lidar data, the radar data, the image data, and the sonar data.
13. The one or more non-transitory computer-readable media of claim 6 , wherein the machine learning model comprises at least one of an attention-based mechanism, a convolutional mechanism, or a multi-scale mechanism.
14. The one or more non-transitory computer-readable media of claim 13 , wherein controlling the vehicle further based on the track comprises:
determining a predicted trajectory for the object based on the track; and
controlling the vehicle further based on the predicted trajectory.
15. A method comprising:
receiving a first sensor observation based on first sensor data representing an environment of a vehicle;
receiving a second sensor observation based on second sensor data representing the environment, wherein the first sensor observation and the second sensor observation represent an object in the environment;
determining, based on a machine learning model, a first similarity matrix including a first similarity value, wherein the first similarity value represents a first similarity of the first sensor observation and the second sensor observation;
determining a track associated with the environment based on the first similarity matrix, wherein the track represents the object;
determining a second similarity matrix that includes a second similarity value, wherein the second similarity value represents a second similarity of a third sensor observation and the track; and
controlling the vehicle based on the second similarity matrix.
16. The method of claim 15 , wherein the first similarity value is determined based on a distance between a first embedding of the first sensor observation and a second embedding of the second sensor observation in an embedding space.
17. The method of claim 15 , wherein determining the track based on the first similarity matrix comprises:
determining that the track is associated with the environment at a previous time.
18. The method of claim 17 , wherein the first similarity matrix is associated with a set of sensor observations comprising the first sensor observation and the second sensor observation, and wherein determining that the track is associated with the environment at the previous time comprises:
determining a set of initial tracks in the environment; and
assigning the set of sensor observations to the set of initial tracks based on the first similarity matrix.
19. The method of claim 18 , wherein the first sensor observation and the second sensor observation are determined to not be associated with any individual tracks.
20. The method of claim 15 , wherein determining that the first sensor observation is not associated with any individual tracks comprises:
determining that the second similarity matrix represents, for the first sensor observation and a second track, a third similarity value, wherein the second track is determined based on sensor observation data associated with a previous time; and
determining, based on the third similarity value, that the first sensor observation is not associated with any individual tracks.