Systems and methods for computer-based labeling of sensor data captured by a vehicle
Examples disclosed herein may involve (i) based on an analysis of 2D data captured by a vehicle while operating in a real-world environment during a window of time, generating a 2D track for at least one object detected in the environment comprising one or more 2D labels representative of the object, (ii) for the object detected in the environment: (a) using the 2D track to identify, within a 3D point cloud representative of the environment, 3D data points associated with the object, and (b) based on the 3D data points, generating a 3D track for the object that comprises one or more 3D labels representative of the object, and (iii) based on the 3D point cloud and the 3D track, generating a time-aggregated, 3D visualization of the environment in which the vehicle was operating during the window of time that includes at least one 3D label for the object.
1. A computer-implemented method comprising:
receiving visualization data that includes a time-aggregated, 3D visualization representing one or more objects in a real-world environment, wherein the time-aggregated, 3D visualization is generated based on a respective set of time-aggregated sensor data corresponding to each respective object of the one or more objects captured by a collection vehicle;
presenting the time-aggregated, 3D visualization on a display associated with a computing system, wherein the time-aggregated, 3D visualization on the display includes 3D data points associated with the one or more objects as a discrete instance of the one or more objects derived from multiple observed instances in the time-aggregated sensor data, the discrete instance selected to represent at least one of the one or more objects at a point along a respective trajectory including the multiple observed instances over the time-aggregated sensor data;
receiving one or more inputs that provide instructions for generating a labeling action on the display for respective 3D data points associated with a given object of the one or more objects, wherein the labeling action is based on the respective set of time-aggregated sensor data of the given object; and
based on the one or more inputs, causing label data for the respective set of time-aggregated sensor data corresponding to the given object to be updated based on the labeling action.
2. The computer-implemented method of claim 1 , wherein the labeling action comprises (i) an indication of a classification of the given object and (ii) a specification of a 3D bounding box for the given object.
3. The computer-implemented method of claim 1 , wherein the visualization data that includes the time-aggregated, 3D visualization comprises previously-generated 3D labels of the one or more objects in the real-world environment, and wherein presenting the time-aggregated, 3D visualization comprises presenting the previously-generated 3D labels of the one or more objects.
4. The computer-implemented method of claim 3 , wherein the labeling action reflects a confirmation of a previously-generated 3D label of the given object.
5. The computer-implemented method of claim 3 , wherein the labeling action reflects a change to a classification of a previously-generated 3D label of the given object.
6. The computer-implemented method of claim 5 , wherein causing the label data for the set of time-aggregated sensor data corresponding to the given object to be updated comprises:
causing a 3D track of the given object to be updated in accordance with the change to the classification of the previously-generated 3D label of the given object.
7. The computer-implemented method of claim 6 , wherein causing the 3D track for the given object to be updated in accordance with the change to the classification of the previously-generated 3D label of the given object comprises propagating the change to the previously-generated 3D label of the given object to each of one or more other 3D labels included in the 3D track of the given object using a motion model.
8. The computer-implemented method of claim 1 , wherein the labeling action comprises an addition of a new 3D label of the given object.
9. The computer-implemented method of claim 1 , wherein the given object was in motion, and wherein a representation of the given object in the time-aggregated, 3D visualization is based on a trajectory of the given object.
10. The computer-implemented method of claim 9 , wherein the trajectory includes multiple data points captured by the collection vehicle.
11. A non-transitory computer-readable medium comprising program instructions stored thereon that are executable to cause a computing system to:
receive visualization data that includes a time-aggregated, 3D visualization representing one or more objects in a real-world environment, wherein the time-aggregated, 3D visualization is generated based on a respective set of time-aggregated sensor data corresponding to each respective object of the one or more objects captured by a collection vehicle;
present the time-aggregated, 3D visualization on a display associated with the computing system, wherein the time-aggregated, 3D visualization on the display includes 3D data points associated with the one or more objects as a discrete instance of the one or more objects derived from multiple observed instances in the time-aggregated sensor data, the discrete instance selected to represent at least one of the one or more objects at a point along a respective trajectory including the multiple observed instances over the time-aggregated sensor data;
receive one or more inputs that provide instructions for generating a labeling action on the display for respective 3D data points associated with a given object of the one or more objects, wherein the labeling action is based on the respective set of time-aggregated sensor data of the given object; and
based on the one or more inputs, cause label data for the respective set of time-aggregated sensor data corresponding to the given object to be updated based on the labeling action.
12. The non-transitory computer-readable medium of claim 11 , wherein the labeling action comprises (i) an indication of a classification of the given object and (ii) a specification of a 3D bounding box for the given object.
13. The non-transitory computer-readable medium of claim 11 , wherein the visualization data that includes the time-aggregated, 3D visualization comprises previously-generated 3D labels of the one or more objects in the real-world environment, and wherein presenting the time-aggregated, 3D visualization comprises presenting the previously-generated 3D labels of the one or more objects.
14. The non-transitory computer-readable medium of claim 13 , wherein the labeling action reflects a confirmation of a previously-generated 3D label of the given object.
15. The non-transitory computer-readable medium of claim 13 , wherein the labeling action reflects a change to a classification of a previously-generated 3D label of the given object.
16. The non-transitory computer-readable medium of claim 15 , wherein causing the label data for the set of time-aggregated sensor data corresponding to the given object to be updated comprises causing a 3D track of the given object to be updated in accordance with the change to the classification of the previously-generated 3D label of the given object.
17. The non-transitory computer-readable medium of claim 16 , wherein causing the 3D track for the given object to be updated in accordance with the change to the classification of the previously-generated 3D label of the given object comprises propagating the change to the previously-generated 3D label of the given object to each of one or more other 3D labels included in the 3D track of the given object using a motion model.
18. The non-transitory computer-readable medium of claim 11 , wherein the labeling action comprises an addition of a new 3D label of the given object.
19. The non-transitory computer-readable medium of claim 11 , wherein the given object was in motion, and wherein a representation of the given object in the time-aggregated, 3D visualization is based on a trajectory of the given object that includes multiple data points captured by the collection vehicle.
20. A computing system comprising:
at least one processor;
a non-transitory computer-readable medium; and
program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing system is capable of:
receiving visualization data that includes a time-aggregated, 3D visualization representing one or more objects in a real-world environment, wherein the time-aggregated, 3D visualization is generated based on a respective set of time-aggregated sensor data corresponding to each respective object of the one or more objects captured by a collection vehicle;
presenting the time-aggregated, 3D visualization on a display associated with the computing system, wherein the time-aggregated, 3D visualization on the display includes 3D data points associated with the one or more objects as a discrete instance of the one or more objects derived from multiple observed instances in the time-aggregated sensor data, the discrete instance selected to represent at least one of the one or more objects at a point along a respective trajectory including the multiple observed instances over the time-aggregated sensor data;
receiving one or more inputs that provide instructions for generating a labeling action on the display for respective 3D data points associated with a given object of the one or more objects, wherein the labeling action is based on the respective set of time-aggregated sensor data of the given object; and
based on the one or more inputs, causing label data for the respective set of time-aggregated sensor data corresponding to the given object to be updated based on the labeling action.