Perception system for an autonomous vehicle
An autonomous platform can obtain sensor data descriptive of an actor in an environment of an autonomous vehicle and at least a portion of the environment of the autonomous vehicle that does not include the actor, the sensor data comprising at least one sweep of the environment of the autonomous vehicle; process the sensor data with a machine-learned perception model to generate a detection of the actor and one or more predicted future velocities; and determine a motion trajectory for the autonomous vehicle based at least in part on the detection and the one or more predicted future velocities.
1. A computer-implemented method, comprising:
obtaining sensor data descriptive of an actor in an environment of an autonomous vehicle and (i) at least a portion of the environment of the autonomous vehicle that does not include the actor, the sensor data comprising at least one sweep of the environment of the autonomous vehicle;
processing the sensor data with a machine-learned perception model to generate a detection of the actor and one or more predicted future velocities;
aligning, by a machine-learned object tracker model, the one or more predicted future velocities to a motion model respective to a class of the actor to generate one or more second velocity outputs;
determining a motion trajectory for the autonomous vehicle based at least in part on the one or more second velocity outputs; and
controlling the autonomous vehicle based at least in part on the motion trajectory.
2. The computer-implemented method of claim 1 , further comprising;
fusing the sensor data from two or more distinct sensor modalities into a common representation of the sensor data,
wherein processing the sensor data is based at least in part on the common representation of the sensor data, and the sensor data is captured from two or more distinct sensor modalities.
3. The computer-implemented method of claim 1 , wherein processing the sensor data further comprises generating, by the machine-learned perception model, one or more uncertainty scores respectively associated with the one or more predicted future velocities.
4. The computer-implemented method of claim 3 , further comprising, prior to determining the motion trajectory for the autonomous vehicle, processing the one or more predicted future velocities and the one or more uncertainty scores with the machine-learned object tracker model configured to generate the one or more second velocity outputs, the second velocity outputs comprising data descriptive of velocities of the actor at one or more discrete future timesteps.
5. The computer-implemented method of claim 4 , wherein the machine-learned object tracker model is configured to smooth the one or more predicted future velocities to generate the one or more second velocity outputs, and wherein the one or more second velocity outputs comprise smoothed velocity outputs.
6. The computer-implemented method of claim 4 , wherein the machine-learned object tracker model comprises a multi-view tracker model and the machine-learned perception model comprises a multi-view perception model.
7. The computer-implemented method of claim 1 , wherein the machine-learned perception model is simultaneously trained to generate the detection of the actor and the one or more predicted future velocities.
8. The computer-implemented method of claim 1 , wherein the one or more predicted future velocities are respectively associated with one or more discrete future time steps.
9. The computer-implemented method of claim 1 , wherein the one or more predicted future velocities are determined in increments up to a prediction end time occurring at a given amount of time after a current time associated with the sensor data.
10. The computer-implemented method of claim 1 , wherein processing the sensor data comprises:
determining bounding box data associated with the actor based on the machine-learned perception model, wherein the machine-learned perception model is configured to regress instantaneous velocities of the actor; and
regressing the one or more predicted future velocities by the machine-learned perception model.
11. The computer-implemented method of claim 1 , wherein the sensor data comprises a plurality of sweeps of the environment of the autonomous vehicle.
12. The computer-implemented method of claim 11 , wherein the sensor data comprises sweep metadata indicative of a relative sweep of the plurality of sweeps in which the sensor data is captured.
13. The computer-implemented method of claim 1 , wherein the machine-learned perception model is trained on training data comprising training sensor data labeled with actual state characteristics of one or more actors depicted in the training sensor data.
14. An autonomous vehicle control system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing executable instructions that cause the one or more processors to perform operations comprising:
obtaining sensor data descriptive of an actor in an environment of an autonomous vehicle and at least a portion of the environment of the autonomous vehicle that does not include the actor, the sensor data comprising at least one sweep of the environment of the autonomous vehicle;
processing the sensor data with a machine-learned perception model to generate a detection of the actor and one or more predicted future velocities;
aligning, by a machine-learned object tracker model, the one or more predicted future velocities to a motion model respective to a class of the actor to generate one or more second velocity outputs;
determining a motion trajectory for the autonomous vehicle based at least in part on the one or more second velocity outputs; and
controlling the autonomous vehicle based at least in part on the motion trajectory.
15. The autonomous vehicle control system of claim 14 , wherein the operations further comprise:
fusing the sensor data from two or more distinct sensor modalities into a common representation of the sensor data,
wherein processing the sensor data is based at least in part on the common representation of the sensor data, and the sensor data is captured from two or more distinct sensor modalities.
16. The autonomous vehicle control system of claim 14 , wherein processing the sensor data further comprises generating, by the machine-learned perception model, one or more uncertainty scores respectively associated with the one or more predicted future velocities.
17. The autonomous vehicle control system of claim 16 , further comprising, prior to determining the motion trajectory for the autonomous vehicle, processing the one or more predicted future velocities and the one or more uncertainty scores with the machine-learned object tracker model configured to generate the one or more second velocity outputs, the second velocity outputs comprising data descriptive of velocities of the actor at one or more discrete future timesteps.
18. An autonomous vehicle, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing executable instructions that cause the one or more processors to perform operations comprising:
obtaining sensor data descriptive of an actor in an environment of the autonomous vehicle and at least a portion of the environment of the autonomous vehicle that does not include the actor, the sensor data comprising at least one sweep of the environment of the autonomous vehicle;
processing the sensor data with a machine-learned perception model to generate a detection of the actor and one or more predicted future velocities;
aligning, by a machine-learned object tracker model, the one or more predicted future velocities to a motion model respective to a class of the actor to generate one or more second velocity outputs;
determining a motion trajectory for the autonomous vehicle based at least in part on the one or more second velocity outputs; and
controlling the autonomous vehicle based at least in part on the motion trajectory.
19. The autonomous vehicle of claim 18 , wherein processing the sensor data further comprises generating, by the machine-learned perception model, one or more uncertainty scores respectively associated with the one or more predicted future velocities.
20. The autonomous vehicle of claim 19 , further comprising, prior to determining a motion trajectory for the autonomous vehicle, processing the one or more predicted future velocities and the one or more uncertainty scores with the machine-learned object tracker model configured to generate the one or more second velocity outputs, the second velocity outputs comprising data descriptive of velocities of the actor at one or more discrete future timesteps.