Cascaded trajectory refinement
Systems and techniques for determining predicted object trajectories are disclosed. A vehicle computing system determines an initial predicted trajectory for an object detected in the environment and determine a predicted location for the object at the endpoint of the trajectory. Using a feature map of the environment, the vehicle computing system refines endpoint location based on a context for the endpoint. The system then determines a middle point of the trajectory based on the refined endpoint location and refines the middle location further based on the context for that location. The system iteratively uses one or more subsequent intermediate points and contexts along the trajectory for further location refinement(s) to generate a refined predicted object trajectory for the object and to determine trajectories for controlling a vehicle.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform operations comprising:
determining a feature map based at least in part on sensor data associated with an environment;
determining, by a machine-learned (ML) model and based at least in part on an initial object location of an object and the feature map, a first predicted object location for the object in the environment;
determining, based at least in part on the feature map and the first predicted object location, a first predicted intermediate object location for the object between the initial object location and the first predicted object location, the first predicted intermediate object location based at least in part on a time horizon of a predicted object trajectory of the object;
determining, based at least in part on the first predicted object location, a first context associated with the first predicted intermediate object location;
determining, by the ML model, based at least in part on the first context and the first predicted intermediate object location, a first refined predicted object location for the object;
determining, based at least in part on the first refined predicted object location and the initial object location, a second predicted intermediate object location for the object in the environment, wherein the second predicted intermediate object location is spatially between the first refined predicted object location and the initial object location;
determining, based at least in part on the feature map and the second predicted intermediate object location, a second context associated with the second predicted intermediate object location;
determining, by the second ML model, based at least in part on the second context and the second predicted intermediate object location, a second refined predicted object location for the object;
determining, based at least in part on the first refined predicted object location and the second refined predicted object location, a refined predicted object trajectory; and
controlling a vehicle based at least in part on the refined predicted object trajectory.
2. The system of claim 1 , wherein the determining the second context is by the second ML model and comprises receiving, as input at the second ML model, one or more of:
the first refined predicted object location;
a third context associated with the first refined predicted object location;
the initial object location; or
a fourth context associated with the initial object location.
3. The system of claim 1 , further comprising a second prediction component in addition to a first prediction component configured to determine the refined predicted object trajectory, wherein the second prediction component is configured to, concurrently with the first prediction component, determine one or more predictions for the object from the initial object location.
4. The system of claim 3 , wherein the second prediction component iteratively determines the one or more predictions wherein individual predictions of the one or more predictions for a first time follows a previous one of the one or more predictions for a second time that is earlier than the first time.
5. The system of claim 1 , wherein controlling the vehicle based at least in part on the refined predicted object trajectory comprises using the refined predicted object trajectory to determine an operational trajectory to control the vehicle to avoid the object in the environment.
6. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
determining a first predicted intermediate object location for an object in an environment, the first predicted intermediate object location being between an initial location and a final location of a predicted object trajectory, the first predicted intermediate object location based at least in part on a time horizon of the predicted object trajectory;
executing a machine-learned (ML) model to determine, based at least in part on the first predicted intermediate object location and a first context associated with the first predicted intermediate object location, a first refined predicted object location for the object;
determining a second predicted intermediate object location for the object in the environment, the second predicted intermediate object location being between the initial location and the first refined predicted object location;
executing the ML model to determine, based at least in part on the first refined predicted object location and a second context associated with the second predicted intermediate object location, a second refined predicted object location for the object;
determining, based at least in part on the first refined predicted object location and the second refined predicted object location, a refined predicted object trajectory; and
controlling a vehicle based at least in part on the refined predicted object trajectory.
7. The one or more non-transitory computer-readable media of claim 6 , wherein the first context is based at least in part on a feature vector in a feature map comprising a top-down representation of the environment determined based at least in part on sensor data associated with the environment.
8. The one or more non-transitory computer-readable media of claim 6 , wherein the first predicted intermediate object location is substantially equidistant from the initial location and the final location.
9. The one or more non-transitory computer-readable media of claim 6 , wherein the final location is predicted by a machine-learned model based on the initial location and a feature map including the initial location.
10. The one or more non-transitory computer-readable media of claim 6 , wherein the second refined predicted object location is determined as being spatially between the initial location and the first refined predicted object location.
11. The one or more non-transitory computer-readable media of claim 6 , wherein the second refined predicted object location is determined as being temporally between the initial location and the first refined predicted object location.
12. The one or more non-transitory computer-readable media of claim 6 , wherein controlling the vehicle comprises:
sending output of the ML model to a vehicle trajectory determination component; and
determining, by the vehicle trajectory determination component and based at least in part on the refined predicted object trajectory, a predicted vehicle trajectory for the vehicle; and
controlling the vehicle based at least in part on the predicted vehicle trajectory.
13. The one or more non-transitory computer-readable media of claim 6 , wherein the first refined predicted object location is associated with a first time that is after a second time associated with the second refined predicted object location.
14. The one or more non-transitory computer-readable media of claim 6 , wherein determining the second predicted intermediate object location comprises using linear interpolation to determine the second predicted intermediate object location based at least in part on the first refined predicted object location and the initial location.
15. A method comprising:
determining, at a vehicle computing system configured at a vehicle in an environment, a first predicted object location for an object in the environment, the first predicted object location associated with a predicted object trajectory;
determining, by the vehicle computing system, based at least in part on the first predicted object location and a first context associated with the first predicted object location, a first refined predicted object location for the object;
determining, by the vehicle computing system, based at least in part on the first refined predicted object location and a current object location, a second predicted object location for the object in the environment, wherein the second predicted object location is between the current object location and the first refined predicted object location;
determining, by the vehicle computing system, based at least in part on the second predicted object location and a second context associated with the second predicted object location, a second refined predicted object location for the object;
determining, based at least in part on the first refined predicted object location and the second refined predicted object location, a refined predicted object trajectory representing a refinement of the predicted object trajectory; and
controlling, by the vehicle computing system and based at least in part on the refined predicted object trajectory, the vehicle.
16. The method of claim 15 , further comprising determining the predicted object trajectory for the object based at least in part on the first refined predicted object location and the second refined predicted object location.
17. The method of claim 15 , wherein determining the second refined predicted object location comprises iteratively determining a plurality of predicted object trajectories by:
iteratively determining a plurality of intermediate predicted object locations;
iteratively determining a plurality of contexts associated with the plurality of intermediate predicted object locations; and
iteratively determining a plurality of refined intermediate predicted object locations based at least in part on the plurality of contexts.
18. The method of claim 15 , wherein the first context is based at least in part on feature data associated with a pixel in a feature map, wherein the pixel is associated with the first predicted object location.
19. The method of claim 15 , wherein the first context is based at least in part on a feature map comprising a top-down representation of the environment determined based at least in part on sensor data associated with the environment.
20. The method of claim 15 , wherein determining second predicted intermediate object location comprises determining, based at least in part on the first refined predicted object location and the current object location, a plurality of intermediate object locations, wherein individual locations of the plurality of intermediate object locations are between the current object location and the first refined predicted object location.