Machine-learned model training for inferring object attributes from distortions in temporal data
Techniques for determining attributes associated with objects represented in temporal sensor data. In some examples, the techniques may include receiving sensor data including a representation of an object in an environment. The sensor data may be generated by a temporal sensor of a vehicle and, in some instances, a trajectory of the object or the vehicle may contribute to a distortion in the representation of the object. For instance, a shape of the representation of the object may be distorted relative to an actual shape of the object. The techniques may also include determining an attribute (e.g., velocity, bounding box, etc.) associated with the object based at least in part on a difference between the representation of the object and another representation of the object (e.g., in other sensor data) or the actual shape of the object.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving sensor data associated with an object in an environment, the sensor data generated by a temporal sensor of a vehicle, wherein a movement of the object or the vehicle contributes to a distortion in a representation of the object in the sensor data relative to an actual shape of the object;
generating time-dimensional sensor data based on the sensor data, the time-dimensional sensor data indicative of the movement of the object through the environment over a period of time;
inputting the time-dimensional sensor data into a machine-learned model;
receiving an output from the machine-learned model, the output including a predicted velocity of the object;
determining a difference between the predicted velocity of the object and a measured velocity of the object; and
based at least in part on the difference meeting or exceeding a threshold difference, altering a parameter of the machine-learned model to minimize the difference and obtain a trained machine-learned model.
2. The system of claim 1 , wherein the time-dimensional sensor data includes a temporal dimension indicating respective points in time at which respective portions of the sensor data were captured by the temporal sensor.
3. The system of claim 1 , wherein the output further includes a predicted bounding box associated with the object, the predicted bounding box indicative of one or more of a size of the object, a location of the object, or an orientation of the object, the operations further comprising:
determining another difference between the predicted bounding box and a ground truth bounding box associated with the object; and
altering at least one of the parameter or another parameter of the machine-learned model to minimize the other difference.
4. The system of claim 1 , wherein the temporal sensor is a rotating lidar sensor, the sensor data is a lidar point cloud, and the generating the time-dimensional sensor data comprises associating respective timestamps with respective points of the lidar point cloud.
5. A method comprising:
receiving sensor data associated with an object;
generating time-dimensional sensor data based on the sensor data, the time-dimensional sensor data indicative of a movement of the object through space over a period of time;
inputting the time-dimensional sensor data into a machine-learned model;
receiving an output from the machine-learned model, the output including a predicted attribute associated with the object;
determining a difference between the predicted attribute and a measured attribute of the object;
based at least in part on the difference meeting or exceeding a threshold difference, altering a parameter of the machine-learned model to minimize the difference and obtain a trained machine-learned model; and
causing the machine-learned model to be sent to a vehicle, the machine-learned model to be used by the vehicle to traverse an environment.
6. The method of claim 5 , wherein the time-dimensional sensor data includes a temporal dimension indicating respective points in time at which respective portions of the sensor data were captured by a temporal sensor.
7. The method of claim 5 , wherein the predicted attribute of the object comprises at least one of a size of the object, a location of the object, an orientation of the object, or a velocity of the object.
8. The method of claim 5 , wherein the predicted attribute of the object comprises a value of a component of a velocity vector associated with the object, the component being perpendicular to a line of sight of a temporal sensor that generated the sensor data.
9. The method of claim 5 , wherein the sensor data is a lidar point cloud generated by a rotating lidar sensor and generating the time-dimensional sensor data comprises associating a respective timestamp with individual points of the lidar point cloud.
10. The method of claim 5 , wherein the sensor data is image data generated by a rolling shutter image sensor and generating the time-dimensional sensor data comprises associating a respective timestamp with a pixel or line of pixels of the image data.
11. The method of claim 5 , wherein the sensor data is generated by a temporal sensor of a vehicle and a trajectory of at least one of the object or the vehicle contributes to a distortion in a representation of the object in the sensor data relative to an actual shape or size of the object.
12. One or more non-transitory computer-readable media storing instruction that, when executed, cause one or more computing device to perform operations comprising:
receiving sensor data associated with an object;
generating time-dimensional sensor data based on the sensor data, the time-dimensional sensor data indicative of a movement of the object through space over a period of time;
inputting the time-dimensional sensor data into a machine-learned model;
receiving an output from the machine-learned model, the output including a predicted attribute associated with the object;
determining a difference between the predicted attribute and a measured attribute of the object;
based at least in part on the difference meeting or exceeding a threshold difference, altering a parameter of the machine-learned model to minimize the difference and obtain a trained machine-learned model; and
causing the machine-learned model to be sent to a vehicle, the machine-learned model to be used by the vehicle to traverse an environment.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the time-dimensional sensor data includes a temporal dimension indicating respective points in time at which respective portions of the sensor data were captured by a temporal sensor.
14. The one or more non-transitory computer-readable media of claim 12 , wherein the predicted attribute of the object comprises at least one of a size of the object, a location of the object, an orientation of the object, or a velocity of the object.
15. The one or more non-transitory computer-readable media of claim 12 , wherein the predicted attribute of the object comprises a value of a component of a velocity vector associated with the object, the component being perpendicular to a line of sight of a temporal sensor that generated the sensor data.
16. The one or more non-transitory computer-readable media of claim 13 , wherein the sensor data is a lidar point cloud generated by a rotating lidar sensor and the generating the time-dimensional sensor data comprises associating a respective timestamp with individual points of the lidar point cloud.
17. The one or more non-transitory computer-readable media of claim 12 , wherein the sensor data is image data generated by a rolling shutter image sensor and the generating the time-dimensional sensor data comprises associating a respective timestamp with a pixel or group of pixels of the image data.
18. The one or more non-transitory computer-readable media of claim 12 , wherein the sensor data is generated by a temporal sensor of a vehicle and a trajectory of at least one of the object or the vehicle contributes to a distortion in a representation of the object in the sensor data relative to an actual shape or size of the object.
19. A method comprising:
receiving sensor data associated with an object, wherein:
the sensor data is generated by a temporal sensor of a vehicle, and
a trajectory of at least one of the object or the vehicle contributes to a distortion in a representation of the object in the sensor data relative to an actual shape or size of the object;
generating time-dimensional sensor data based on the sensor data, the time-dimensional sensor data indicative of a movement of the object through space over a period of time;
inputting the time-dimensional sensor data into a machine-learned model;
receiving an output from the machine-learned model, the output including a predicted attribute associated with the object;
determining a difference between the predicted attribute and a measured attribute of the object; and
based at least in part on the difference meeting or exceeding a threshold difference, altering a parameter of the machine-learned model to minimize the difference and obtain a trained machine-learned model.
20. One or more non-transitory computer-readable media storing instruction that, when executed, cause one or more computing device to perform operations comprising:
receiving sensor data associated with an object, wherein:
the sensor data is generated by a temporal sensor of a vehicle, and
a trajectory of at least one of the object or the vehicle contributes to a distortion in a representation of the object in the sensor data relative to an actual shape or size of the object;
generating time-dimensional sensor data based on the sensor data, the time-dimensional sensor data indicative of a movement of the object through space over a period of time;
inputting the time-dimensional sensor data into a machine-learned model;
receiving an output from the machine-learned model, the output including a predicted attribute associated with the object;
determining a difference between the predicted attribute and a measured attribute of the object; and
based at least in part on the difference meeting or exceeding a threshold difference, altering a parameter of the machine-learned model to minimize the difference and obtain a trained machine-learned model.