Flexible hardware design for camera calibration and image pre-procesing in autonomous driving vehicles
A sensor unit includes a sensor interface, host interface, and pre-processing hardware. The sensor interface is coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV). The host interface is coupled to a perception and planning system. The pre-processing hardware is coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images to the perception and planning system via the host interface. The perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment. The pre-processing functions can adjust for different calibrations and formats across the plurality of cameras.
1. A sensor unit, comprising:
a sensor interface to be coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV);
a host interface to be coupled to a perception and planning system; and
pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to the perception and planning system via the host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
2. The sensor unit of claim 1 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
3. The sensor unit of claim 1 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
4. The sensor unit of claim 3 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
5. The sensor unit of claim 4 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
6. The sensor unit of claim 1 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
7. The sensor unit of claim 6 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.
8. A computer-implemented method for pre-processing images for autonomous driving, the method comprising:
receiving, at a sensor unit via a sensor interface, a plurality of images from a plurality of cameras around an autonomous driving vehicle (ADV);
performing one or more pre-processing functions on the images by pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry; and
transmitting pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to a perception and planning system via a host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on at least the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
9. The method of claim 8 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
10. The method of claim 8 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
11. The method of claim 10 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
12. The method of claim 11 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
13. The method of claim 8 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
14. The method of claim 13 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.
15. An autonomous driving system, comprising:
a plurality of sensors, the sensors including one or more cameras to capture images around an autonomous driving vehicle (ADV);
a perception and planning system; and
a sensor unit, wherein the sensor unit includes
a sensor interface coupled to the sensors,
a host interface coupled to the perception and planning system, and
pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry coupled to the sensor interface to receive images from the cameras, to perform one or more pre-processing functions on the images, and to transmit pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to the perception and planning system via the host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on at least the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
16. The system of claim 15 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
17. The system of claim 15 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
18. The system of claim 17 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
19. The system of claim 18 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
20. The system of claim 15 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
21. The system of claim 20 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.