Lidar and camera data fusion for automated vehicle
View Patent ↗A data-fusion system that fuses lidar-data and camera-data for an automated vehicle includes a camera, a lidar, and a controller. The camera renders an image of an object proximate to a host-vehicle. The lidar detects a distance and a direction to the object based on a reflected-signal of light reflected by the object. The controller is in communication with the camera and the lidar. The controller is configured to determine a reflectivity-characteristic of the object based on the image and the reflected-signal, and adjust a detection-characteristic of the lidar when the reflectivity-characteristic of the object makes it difficult for the lidar to detect the distance and the direction to the object.
1. A system comprising:
a camera of a host-vehicle that renders an image of an object proximate to the host-vehicle;
a light detection and ranging (lidar) sensor of the host-vehicle that transmits light-energy that impinges the object;
a controller in communication with the camera and the lidar, said controller configured to:
determine, using the image, a reflection-fraction of the light-energy that impinged on the object and is reflected by the object towards the lidar;
compare the reflection-fraction with a fraction-threshold; and
in accordance with a result of the comparing, adjust a detection-characteristic of the lidar.
2. The system in accordance with claim 1 , wherein the reflection-fraction is also determined based on a signal-strength of the reflected light-energy detected by the lidar.
3. The system in accordance with claim 1 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is characterized as non-reflective.
4. The system in accordance with claim 1 , wherein the reflection-fraction is less than the fraction-threshold when a color of the object is black.
5. The system in accordance with claim 1 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is characterized as highly-reflective.
6. The system in accordance with claim 1 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is chrome.
7. The system in accordance with claim 1 , wherein the detection-characteristic includes a detector-gain of the lidar, and the detector-gain is increased when the reflection-fraction is less than the fraction-threshold.
8. The system in accordance with claim 1 , wherein the detection-characteristic includes a detection-threshold of the lidar, and the detection-threshold is decreased when the reflection-fraction is less than the fraction-threshold.
9. The system in accordance with claim 1 , wherein the reflection-fraction is determined using a strength of the reflected light-energy detected in the image.
10. The system in accordance with claim 1 , wherein the fraction threshold is adjusted according to environmental conditions.
11. The system in accordance with claim 1 , wherein adjusting the detection-characteristic of the lidar includes increasing a detector-gain or a detector-threshold of the lidar to better detect a distance or direction to the object.
12. The system in accordance with claim 1 , wherein adjusting the detection-characteristic of the lidar includes adjusting a portion of a field-of-view of the lidar that corresponds to a bounding-box that is placed around the object by an object-detector.
13. A method comprising:
rendering, using a camera of a host-vehicle, an image of an object proximate to the host-vehicle;
transmitting, by a light detection and ranging (lidar) sensor of the host-vehicle, light-energy that impinges the object;
determining, by a controller of the host-vehicle, a reflection-fraction of the light-energy that impinged on the object and is reflected by the object towards the lidar, the reflection-fraction determined using the image;
comparing, by the controller, the reflection-fraction with a fraction-threshold; and
in accordance with the reflection-fraction being less than the fraction-threshold,
adjusting a detection-characteristic of the lidar.
14. The method in accordance with claim 13 , wherein the reflection-fraction is also determined based on a signal-strength of the reflected light-energy.
15. The method in accordance with claim 13 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is characterized as non-reflective.
16. The method in accordance with claim 13 , wherein the reflection-fraction is less than the fraction-threshold when a color of the object is black.
17. The method in accordance with claim 13 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is characterized as highly-reflective.
18. The method in accordance with claim 13 , wherein the reflection-fraction is less than the fraction-threshold when a surface of the object is chrome.
19. The method in accordance with claim 13 , wherein the detection-characteristic includes a detector-gain of the lidar, and the detector-gain is increased when the reflection-fraction is less than the fraction-threshold.
20. The method in accordance with claim 13 , wherein the detection-characteristic includes a detection-threshold of the lidar, and the detection-threshold is decreased when the reflection-fraction is less than the fraction-threshold.
21. The method in accordance with claim 13 , wherein the reflection-fraction is determined using a strength of the reflected light-energy detected in the image.
22. The method in accordance with claim 13 , wherein the fraction threshold is adjusted according to environmental conditions.
23. The method in accordance with claim 13 , wherein adjusting the detection-characteristic of the lidar includes increasing a detector-gain or a detector-threshold of the lidar to better detect a distance or direction to the object.
24. The method in accordance with claim 13 , wherein adjusting the detection-characteristic of the lidar includes adjusting a portion of a field-of-view of the lidar that corresponds to a bounding-box that is placed around the object by an object-detector.