Merging LiDAR information and camera information
Among other things, techniques are described for merging LiDAR information and camera information for autonomous annotation. The techniques include a vehicle that includes at least one LiDAR device configured to detect electromagnetic radiation; at least one camera configured to generate camera information of objects proximate to the vehicle; at least one computer-readable media storing computer-executable instructions; at least one processor communicatively coupled to the at least one LiDAR device and the at least one camera and a control circuit communicatively coupled to the at least one processor, wherein the control circuit is configured to operate the vehicle based on a location of the object.
1. A vehicle, comprising:
at least one non-transitory computer-readable medium storing computer-executable instructions;
at least one processor communicatively coupled to the at least one non-transitory computer-readable medium, the at least one processor configured to execute the computer executable instructions, the execution carrying out operations including:
receiving LiDAR information from at least one LiDAR device;
receiving object information of an object proximate to the vehicle based on camera information from at least one camera, the object information comprising at least one pixel of the camera information representing the object and the object information comprising categorization information representing a classification of the object that indicates a category or a type of the object;
merging at least one portion of the received LiDAR information with at least one pixel associated with the received object information to generate merged information representing the object;
determining a location of the object relative to the vehicle based on the merged information representing the object or a combination of the LiDAR information and the camera information;
determining an accuracy of the object based on at least one of a number of pixels associated with the object, the location of the object relative to the vehicle, or the merged information representing the object; and
determining a confidence level of the merged information actually representing the object based on the accuracy of the object, the confidence level representing how confident the vehicle is that the object proximate to the vehicle is in fact the object actually represented by the merged information; and
a control circuit communicatively coupled to the at least one processor, the control circuit configured to operate the vehicle based on (i) the location of the object relative to the vehicle and (ii) the confidence level of the merged information actually representing the object.
2. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: filtering the merged information based on the categorization information.
3. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: determining an orientation of the object based on the merged information.
4. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including determining if the categorization information of the object is associated with a traffic instruction, wherein the traffic instruction is a traffic light or a traffic sign.
5. The vehicle of claim 4 , wherein the at least one processor is further configured to carry out operations including determining a traffic signal of the traffic instruction based on the merged information.
6. The vehicle of claim 5 , wherein the control circuit is further configured to operate the vehicle based on the traffic signal of the traffic instruction.
7. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: assigning an instance identifier to the object based on the merged information.
8. The vehicle of claim 7 , wherein the at least one processor is further configured to carry out operations including:
determining if the object represents two distinct objects based on the merged information; and
in accordance with determining if the object represents two distinct objects, assigning a unique instance identifier to each of the two distinct objects.
9. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: annotating a map based on the merged information.
10. The vehicle of claim 9 , wherein annotating a map comprises transmitting the location of the object, the categorization information of the object, an instance identifier of the object, and a date of observing the object to a database hosting the map.
11. The vehicle of claim 10 , wherein annotating the map further comprises:
transmitting at least one of a number of times that the object has been observed, a frequency that the object has been observed, or a date and time of observing the object to the database for determining the confidence level of the merged information actually representing the object.
12. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: updating an existing instance of the object on a map based on the merged information.
13. The vehicle of claim 1 , wherein the at least one processor is further configured to carry out operations including: determining at least one geometric feature of the object based on the merged information.
14. The vehicle of claim 13 , wherein determining the at least one geometric feature of the object comprises determining one of at least one edge of the object, at least one surface of the object, or size of the object.
15. The vehicle of claim 1 , wherein the at least one camera acquires the camera information and the at least one LiDAR device acquires the LiDAR information concurrently.
16. The vehicle of claim 1 , wherein the at least one camera acquires the camera information after the at least one LiDAR device acquires the LiDAR information.
17. The vehicle of claim 16 , wherein a timing difference between when the at least one camera acquires the camera information and the at least one LiDAR device acquires the LiDAR information is based on a velocity of the vehicle.
18. The vehicle of claim 1 , wherein merging the at least one portion of the received LiDAR information with the at least one pixel associated with the received object information comprises merging a plurality of LiDAR points within a bounding box of the camera information.
19. The vehicle of claim 1 , wherein determining the confidence level of the merged information actually representing the object comprises determining the confidence level further based on additional information that comprises at least one of:
a number of times the object has been observed,
a number of times the object has been observed within a predetermined number of days,
a date and time of observing the object, or
an observation frequency of the object.
20. The vehicle of claim 1 , wherein the operations further comprise:
in response to determine that at least one of the accuracy or the confidence level is below a corresponding threshold, determining that a second look is warranted and controlling or instructing to control at least one of:
a zoom feature of the at least one camera or a second camera system to zoom-in on the object,
panning an area of the object based on at least one of the camera information or the merged information,
a zoom feature of the at least one LiDAR device or a second LiDAR system to zoom-in on the object, or
panning an area of the object based on at least one of the LiDAR information or the merged information.
21. A method comprising:
receiving LiDAR information from at least one LiDAR device of a vehicle;
receiving object information of an object proximate to the vehicle based on camera information from at least one camera, the object information comprising at least one pixel of the camera information representing the object and the object information comprising categorization information representing a classification of the object that indicates a category or a type of the object;
determining if the categorization information of the object is associated with a traffic instruction; and
in accordance with determining that the categorization information of the object is associated with the traffic instruction:
merging at least one portion of the received LiDAR information with at least one pixel associated with the received object information to generate merged information representing the object;
determining a location of the object relative to the vehicle based on the merged information representing the object or a combination of the LiDAR information and the camera information;
determining an accuracy of the object based on at least one of a number of pixels associated with the object, the location of the object relative to the vehicle, or the merged information representing the object;
determining a confidence level of the merged information actually representing the object based on the accuracy of the object, the confidence level representing how confident the vehicle is that the object proximate to the vehicle is in fact the object actually represented by the merged information; and
operating the vehicle based on (i) the location of the object relative to the vehicle and (ii) the confidence level of the merged information actually representing the object.
22. A non-transitory computer-readable storage medium comprising at least one program for execution by at least one processor of a first device, the at least one program including instructions which, when executed by at least one processor, cause the first device to perform operations comprising:
receiving LiDAR information from at least one LiDAR device of a vehicle;
receiving object information of an object proximate to the vehicle based on camera information from at least one camera, the object information comprising at least one pixel of the camera information representing the object and the object information comprising categorization information representing a classification of the object that indicates a category or a type of the object;
determining if the categorization information of the object is associated with a traffic instruction; and
in accordance with determining that the categorization information of the object is associated with the traffic instruction:
merging at least one portion of the received LiDAR information with at least one pixel associated with the received object information to generate merged information representing the object;
determining a location of the object relative to the vehicle based on the merged information representing the object or a combination of the LiDAR information and the camera information;
determining an accuracy of the object based on at least one of a number of pixels associated with the object, the location of the object relative to the vehicle, or the merged information representing the object;
determining a confidence level of the merged information actually representing the object based on the accuracy of the object, the confidence level representing how confident the vehicle is that the object proximate to the vehicle is in fact the object actually represented by the merged information; and
operating the vehicle based on (i) the location of the object relative to the vehicle and (ii) the confidence level of the merged information actually representing the object.