Object collision path prediction
System, methods, and computer-readable media for an object path prediction model, and an associated training technique, to output a path that is considered an object collision path. The object path prediction model outputs a set of predicted paths for the object that are outputted to a trained planning algorithm, which includes paths that are most likely to occur and a path that is considered an object collision path. The predicted paths are sent to the trained planning algorithm and used for planning a trajectory for the autonomous vehicle that is associated with a low probability of colliding with or taking a sudden evasive action to avoid the object.
1. A method for identifying an object collision path, the method comprising:
operating an autonomous vehicle (AV) having a local computing device configured to execute a prediction stack and a planning stack, wherein the planning stack is configured to determine how to maneuver the AV within the environment:
receiving, by an object path prediction algorithm, sensor data indicative of objects in an environment, the sensor data being generated by sensors of the AV;
receiving, by the object path prediction algorithm, information about a location of the autonomous vehicle in the environment;
determining, by the object path prediction algorithm, a set of predicted paths for an object from the objects in the environment based on the sensor data;
outputting, from the object path prediction algorithm, selected paths from the set of predicted paths for the object, wherein the object path prediction algorithm is configured to output, to a planning algorithm, a fixed number of paths that are most likely to occur and a path that is considered an object collision path, the object collision path being a path that if taken by the object would result in the AV taking evasive action to avoid the object, wherein the object path prediction algorithm is a multimodal machine-learning prediction algorithm with a dedicated object collision head;
outputting, via the dedicated object collision head, the object collision path to the planning stack of the AV; and
executing the planning stack to control a maneuver of the AV based on the output object collision path.
2. The method of claim 1 , wherein the multimodal machine-learning prediction algorithm further comprises a multi-class classification head, the method further comprising:
outputting, via the multi-class classification head, probabilities associated with each mode of the multimodal machine-learning prediction algorithm, wherein the probabilities have a sum of 1.
3. The method of claim 2 , wherein the multimodal machine-learning prediction algorithm further comprises a binary classification head, the method further comprising:
outputting, via the binary classification head, a binary label characterizing whether the object collision path is relevant or not to the planning algorithm.
4. The method of claim 3 , wherein the planning algorithm uses the binary classification output to calibrate the relevancy of the object collision path.
5. The method of claim 1 , further comprising:
sending the selected paths including the object collision path to the planning algorithm for use in planning a trajectory for the autonomous vehicle that is associated with a low probability of colliding with or taking a sudden evasive action to avoid the object.
6. The method of claim 1 , wherein the object path prediction algorithm is trained by being provided a labeled dataset including the sensor data, and subsequent trajectories observed to be traveled by the objects in the environment, wherein the subsequent trajectories are labeled to identify when a path traveled by the object was an object collision path.
7. The method of claim 6 , wherein the labeled dataset is at least partially derived from road event data collected by the autonomous vehicle that includes a flag to indicate that an autonomous vehicle technical operator has taken over control of the autonomous vehicle, and the labeled dataset has labels to identify when the path traveled by the object was the object collision path.
8. A system for identifying an object collision path from a trained object path prediction algorithm, comprising:
a storage configured to store instructions;
a processor configured to execute the instructions and cause the processor to:
operate an autonomous vehicle (AV), wherein the processor is configured to execute a prediction stack and a planning stack, wherein the planning stack is configured to determine how to maneuver the AV within the environment:
receive, by an object path prediction algorithm of the prediction stack, sensor data indicative of objects in an environment, the sensor data being generated by sensors of the AV;
receive by the object path prediction algorithm, information about a location of the autonomous vehicle in the environment;
determine, by the object path prediction algorithm, a set of predicted paths for an object from the objects in the environment;
output, from the object path prediction algorithm, selected paths from the set of predicted paths for the object, wherein the object path prediction algorithm is configured to output, to the planning stack, a fixed number of paths that are most likely to occur and a path that is considered an object collision path, the object collision path being a path that if taken by the object would result in the AV taking evasive action to avoid the object, wherein the object path prediction algorithm is a multimodal machine-learning prediction algorithm with a dedicated object collision head, wherein the processor is configured to execute the instructions and cause the processor to: output, via the dedicated object collision head, the object collision path; and control a maneuver of the AV based on the output object collision path.
9. The system of claim 8 , wherein the multimodal machine-learning prediction algorithm further comprises a multi-class classification head, wherein the processor is configured to execute the instructions and cause the processor to:
output, via the multi-class classification head, probabilities associated with each mode of the multimodal machine-learning prediction algorithm, wherein the probabilities have a sum of 1.
10. The system of claim 9 , wherein the multimodal machine-learning prediction algorithm further comprises a binary classification head, wherein the processor is configured to execute the instructions and cause the processor to:
output, via the binary classification head, a binary label characterizing whether the object collision path is relevant or not to the planning algorithm.
11. The system of claim 10 , wherein the planning algorithm uses the binary classification output to calibrate the relevancy of the object collision path.
12. The system of claim 8 , wherein the processor is configured to execute the instructions and cause the processor to:
send the selected paths including the object collision path to the planning algorithm for use in planning a trajectory for the autonomous vehicle that is associated with a low probability of colliding with or taking a sudden evasive action to avoid the object.
13. The system of claim 8 , wherein the object path prediction algorithm is trained by being provided a labeled dataset including the information about the objects in the environment as perceived by the sensors of the autonomous vehicle, and subsequent trajectories observed to be traveled by the objects in the environment, wherein the subsequent trajectories are labeled to identify when a path traveled by the object was an object collision path.
14. The system of claim 13 , wherein the labeled dataset is at least partially derived from road event data collected by the autonomous vehicle that includes a flag to indicate that an autonomous vehicle technical operator has taken over control of the autonomous vehicle, and the labeled dataset has labels to identify when the path traveled by the object was the object collision path.
15. A non-transitory computer-readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:
operate an autonomous vehicle (AV), wherein the computing system is configured to execute a prediction stack and a planning stack, wherein the planning stack is configured to determine how to maneuver the AV within the environment:
receive, by an object path prediction algorithm of the prediction stack, sensor data indicative of objects in an environment, the sensor data being generated by sensors of the AV;
receive, by the object path prediction algorithm, information about a location of the autonomous vehicle in the environment;
determine, by the object path prediction algorithm, a set of predicted paths for an object from the objects in the environment;
output, from the object path prediction algorithm, selected paths from the set of predicted paths for the object, wherein the object path prediction algorithm is configured to output, to the planning stack, a fixed number of paths that are most likely to occur and a path that is considered an object collision path, the object collision path being a path that if taken by the object would result in the AV taking evasive action to avoid the object, wherein the object path prediction algorithm is a multimodal machine-learning prediction algorithm with a dedicated object collision head;
outputting, via the dedicated object collision head, the object collision path; and controlling a maneuver of the AV based on the output object collision path.
16. The non-transitory computer-readable medium of claim 15 , wherein the multimodal machine-learning prediction algorithm further comprises a multi-class classification head, wherein the instructions further caused to the computing system to:
output, via the multi-class classification head, probabilities associated with each mode of the multimodal machine-learning prediction algorithm, wherein the probabilities have a sum of 1.
17. The non-transitory computer-readable medium of claim 16 , wherein the multimodal machine-learning prediction algorithm further comprises a binary classification head, wherein the instructions further caused to the computing system to:
output, via the binary classification head, a binary label characterizing whether the object collision path is relevant or not to the planning algorithm.