Object identification and labeling tool for training autonomous vehicle controllers
Techniques for identifying and labeling distinct objects within 3-D images of environments in which vehicles operate, to thereby generate training data used to train models that autonomously control and/or operate vehicles, are disclosed. A 3-D image may be presented from various perspective views (in some cases, dynamically), and/or may be presented with a corresponding 2-D environment image in a side-by-side and/or a layered manner, thereby allowing a user to more accurately identify groups/clusters of data points within the 3-D image that represent distinct objects. Automatic identification/delineation of various types of objects depicted within 3-D images, automatic labeling of identified/delineated objects, and automatic tracking of objects across various frames of a 3-D video are disclosed. A user may modify and/or refine any automatically generated information. Further, at least some of the techniques described herein are equally applicable to 2-D images.
1. A computer-implemented method for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the method comprising:
displaying, on a user interface of one or more computing devices, a three-dimensional (3-D) image of an environment in which vehicles operate, the 3-D image depicting one or more physical objects located in the environment;
providing, on the user interface, a paint user control for use by a user to indicate areas within images displayed on the user interface;
receiving, via a user activation of the paint user control, an indication of a data point within the 3-D image;
based upon the indicated data point within the 3-D image,
(i) automatically determining, without any additional user input aside from the user activation, boundaries of a depiction of an essentially planar surface area within the 3-D image, the depiction of the essentially planar surface area including the indicated data point and other data points surrounding the indicated data point within the 3-D image, and the essentially planar surface area being a particular type of surface area,
(ii) automatically determining, from a plurality of visual properties, each of which denotes a different type of surface area, a particular visual property corresponding to the particular type of the essentially planar surface area, and
(iii) automatically modifying, by the one or more computing devices and based on the determined boundaries, the particular visual property of the data points included within the boundaries of the depiction of the essentially planar surface area of the particular type;
obtaining, by the one or more computing devices, an indication of a particular label for the automatically determined depiction of the essentially planar surface area of the 3-D image; and
storing, by the one or more computing devices in one or more tangible, non-transitory memories, an indication of an association between data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label, thereby distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image.
2. The method of claim 1 , wherein the particular label is included in a plurality of labels of a plurality of areas and/or physical objects, and wherein each label included in the plurality of labels corresponds to a respective visual property.
3. The computer-implemented method of claim 1 , wherein the user activation of the paint user control consists of only a single mouse click.
4. The computer-implemented method of claim 1 , wherein the automatically determined depiction of the essentially planar surface area within the 3-D image is a first area representing a first essentially planar surface area of a first particular type, and the visual property is a first visual property, and the method further comprises modifying a second area within the 3-D image with a second visual property different than the first visual property, the second area being another subset of the total area of the 3-D image representing a second essentially planar surface area of a second particular type, and the modifying of the second area based on a usage of the paint user control as a virtual paint brush, a virtual paint spill, or another type of usage of the paint user control.
5. The computer-implemented method of claim 4 , wherein the usage of the paint user control is as the virtual paint brush, and modifying the second area with the second visual property comprises the user manipulating the virtual paint brush across the second area within the 3-D image.
6. The computer-implemented method of claim 5 , wherein a width of the virtual paintbrush is modifiable.
7. The computer-implemented method of claim 1 , wherein receiving, via the user activation of the paint user control, the indication of the data point within the 3-D image comprises receiving an indication of the user activation of the paint user control while a cursor or electronic pointer controlled by the paint user control hovers over or points to a location of the data point.
8. The computer-implemented method of claim 1 , further comprising receiving an indication of a particular type of object depicted in the 3-D image, the particular type of object corresponding to the essentially planar surface area.
9. The computer-implemented method of claim 1 , wherein the essentially planar surface area includes a road.
10. The computer-implemented method of claim 1 , wherein the essentially planar surface area includes a ground.
11. The computer-implemented method of claim 1 , wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises automatically applying the particular visual property to the data points included within the depiction of the essentially planar surface area.
12. The computer-implemented method of claim 11 , further comprising receiving, via the paint user control, an indication of a portion of the data points included within the depiction of the essentially planar surface area from which the particular visual property is to be removed.
13. The computer-implemented method of claim 1 , wherein automatically modifying the particular visual property of the data points included within the depiction of the essentially planar surface area comprises removing the particular visual property from the data points included within the depiction of the essentially planar surface area of the 3-D image.
14. The computer-implemented method of claim 1 , wherein automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image comprises automatically determining the boundaries of the depiction of the essentially planar surface area within the 3-D image by utilizing an object identification model that has been trained based on identified objects depicted within a plurality of historical images of one or more environments in which vehicles operate.
15. The computer-implemented method of claim 1 , wherein 3-D image of the environment comprises a dataset generated by one or more active sensing devices or systems.
16. The computer-implemented method of claim 15 , wherein the one or more active sensing devices or systems include one or more light detection and ranging (LIDAR) devices.
17. The computer-implemented method of claim 15 , wherein dataset is a point cloud dataset.
18. The computer-implemented method of claim 1 , wherein the 3-D environment image is an interactive, three-dimensional (3-D) image of the environment.
19. The computer-implemented method of claim 1 , wherein storing the indication of the association between the data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label comprises storing data indicative of the particular label in association with each pixel or data point included in the automatically determined depiction of the essentially planar surface area of the 3-D image.
20. The computer-implemented method of claim 1 , wherein the method further comprises:
receiving, via the user interface, an instruction to hide respective images of one or more specific objects depicted within the 3-D image of the environment; and
based on the received hiding instruction, rendering non-visible the respective images of the one or more specific objects depicted within the 3-D image of the environment while maintaining respective levels of visibility of other objects and/or areas within the 3-D image of the environment, thereby preventing the respective images of the one or more specific objects from being indicated via the paint user control.
21. The computer-implemented method of claim 1 , further comprising providing the stored data indicative of the association between the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label as training data for a machine-learning model that generates control signals to autonomously control one or more autonomous operations of a vehicle.
22. The computer-implemented method of claim 1 , wherein receiving the indication of the particular label is executed prior to receiving the indication of the data point within the 3-D image.
23. The computer-implemented method of claim 1 , wherein receiving the indication of the data point within the 3-D image is executed prior to receiving the indication of the particular label.
24. A system for identifying and labeling objects within images for training machine-learning based models that are used to autonomously operate vehicles, the system comprising:
a communication module;
one or more processors; and
one or more non-transitory, tangible memories coupled to the one or more processors and storing computer executable instructions thereon that, when executed by the one or more processors, cause the system to:
display, on a user interface, a three-dimensional (3-D) image of an environment in which vehicles operate, the 3-D image depicting one or more physical objects located in the environment;
provide, on the user interface, a paint user control for use by a user to indicate areas within the 3-D image;
receive, via the communication module, an indication of a user activation of the paint user control at a location of a data point within the 3-D image;
based upon the indicated location of the data point, (i) automatically determine, without any additional user input aside from the user activation, boundaries of a depiction of an essentially planar surface area within the 3-D image, the depiction of the essentially planar surface area including the indicated data point and other data points surrounding the indicated data point within the 3-D image, and the essentially planar surface area being a particular type of surface area, (ii) automatically determine, from a plurality of visual properties, each of which denotes a different type of surface area, a particular visual property corresponding to the particular type of the essentially planar surface area, and (iii) automatically modify, by the one or more computing devices and based on the determined boundaries, the particular visual property of the data points included within the boundaries of the depiction of the essentially planar surface area of the particular type;
obtain an indication of a particular label for the automatically determined depiction of the essentially planar surface area; and
store, in the one or more tangible, non-transitory memories, an indication of an association between data indicative of the automatically determined depiction of the essentially planar surface area of the 3-D image and the particular label, thereby distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image.
25. The system of claim 24 , wherein the particular label is included in a plurality of labels of a plurality of areas and/or objects, and wherein each label included in the plurality of labels corresponds to a respective visual property.
26. The system of claim 24 , wherein the automatic determination of the depiction of the essentially planar surface area within the 3-D image comprises a utilization of an object identification model that has been trained based on identified objects depicted within a plurality of historical images of one or more environments in which vehicles operate.
27. The system of claim 24 , wherein the stored association distinguishing the essentially planar surface area from other physical objects depicted within the 3-D image is included in a set of training data for a machine-learning model that generates control signals to autonomously control one or more operations of a vehicle.