Automatic labeling of objects in sensor data
Aspects of the disclosure provide for automatically generating labels for sensor data. For instance, first sensor data for a vehicle may be identified. This first sensor data may have been captured by a first sensor of the vehicle at a first location during a first point in time and may be associated with a first label for an object. Second sensor data for the vehicle may be identified. The second sensor data may have been captured by a second sensor of the vehicle at a second location at a second point in time outside of the first point in time. The second location is different from the first location. A determination may be made as to whether the object is a static object. Based on the determination that the object is a static object, the first label may be used to automatically generate a second label for the second sensor data.
1 . A method comprising:
identifying, by one or more processors, first sensor data for a vehicle, wherein the first sensor data is captured by a first sensor of the vehicle (i) at a first location, (ii) at a first point in time, and (iii) within a finite period of time or a timeframe, and wherein the first point in time is based at least on GPS timing signals;
identifying, by the one or more processors, second sensor data for the vehicle, wherein the second sensor data is captured by a second sensor of the vehicle (i) at a second location and (ii) at a second point in time that is different than the first point in time; and
associating, by the one or more processors, a label for an object with the second sensor data, wherein the label is based on the first sensor data.
2 . The method of claim 1 , wherein the sensor is a laser-based sensor and the second sensor is a camera.
3 . The method of claim 1 , wherein the label is a three-dimensional bounding box identifying a location for the object in the first sensor data.
4 . The method of claim 1 , wherein the second sensor data includes a set of camera images, and the method further comprises filtering the set of camera images to remove camera images that do not include the object, and wherein associating the label includes associating the label with one or more camera images of the filtered set of camera images.
5 . The method of claim 1 , wherein the second sensor data includes a set of camera images, and the method further comprises:
projecting the label into each camera image of the set of camera images; and
subsequent to projecting the label, filtering the set of camera images to remove camera images where the projected label is smaller than a given size relative to the respective camera image,
wherein associating the label includes associating the label with one or more camera images of the filtered set of camera images.
6 . The method of claim 1 , wherein the second location is beyond an effective perceptive range of the second sensor due to a weather condition when the vehicle is at the second location, and the object is not in the second sensor data.
7 . The method of claim 1 , wherein the second location is beyond an effective perceptive range of the second sensor due to another object occluding the second location when the vehicle is at the second location, and the object is not in the second sensor data.
8 . The method of claim 1 , wherein the second location is beyond a maximum perceptive range of the second sensor when the vehicle is at the second location, and the object is not in the second sensor data.
9 . The method of claim 1 , wherein the label identifies a location for the object that is beyond an effective perceptive range of the first sensor when the vehicle is at the second location.
10 . The method of claim 1 , wherein the label identifies a location for the object that is beyond a maximum perceptive range of the first sensor when the vehicle is at the second location.
11 . The method of claim 1 , further comprising, prior to associating the label, determining that the object is not occluded with respect to the second sensor at the second point in time.
12 . The method of claim 11 , wherein determining that the object is not occluded includes building a surfel map and casting a ray from the vehicle to a location of the object identified in the object in the second sensor data.
13 . The method of claim 11 , wherein determining that the object is not occluded includes:
providing for display a portion of the first sensor data with a camera image of the second sensor data, wherein the camera image includes a three-dimensional bounding box for the label projected into two-dimensional space of the camera image; and
receiving confirmation from a human operator that the object is not occluded.
14 . The method of claim 11 , wherein determining that the object is not occluded includes inputting the label and a camera image of the second sensor data into a machine-learned model.
15 . The method of claim 1 , wherein the first point in time and the second point in time are at least 0.5 second apart from one another.
16 . A system comprising one or more processors configured to:
identify first sensor data for a vehicle, wherein the first sensor data is captured by a first sensor of the vehicle (i) at a first location, (ii) at a first point in time, and (iii) within a finite period of time or a timeframe, and wherein the first point in time is based at least on GPS timing signals;
identify second sensor data for the vehicle, wherein the second sensor data is captured by a second sensor of the vehicle (i) at a second location and (ii) at a second point in time that is different than the first point in time; and
associate a label for an object with the second sensor data, wherein the label is based on the first sensor data.
17 . The system of claim 16 , wherein the second location is beyond an effective perceptive range of the second sensor due to a weather condition when the vehicle is at the second location, and the object is not in the second sensor data.
18 . The system of claim 16 , wherein the second location is beyond an effective perceptive range of the second sensor due to another object occluding the second location when the vehicle is at the second location, and the object is not in the second sensor data.
19 . The system of claim 16 , wherein the second location is beyond a maximum perceptive range of the second sensor when the vehicle is at the second location, such thatand the object is not detected in the second sensor data but is associated with the second label.
20 . A non-transitory, tangible, computer-readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to implement a method, the method comprising:
identifying first sensor data for a vehicle, wherein the first sensor data is captured by a first sensor of the vehicle (i) at a first location, (ii) at a first point in time, and (iii) within a finite period of time or a timeframe, and wherein the first point in time is based at least on GPS timing signals;
identifying second sensor data for the vehicle, wherein the second sensor data is captured by a second sensor of the vehicle (i) at a second location and (ii) at a second point in time that is different than the first point in time; and
associating a label for an object with the second sensor data, wherein the label is based on the first sensor data.