System and apparatus for co-registration and correlation between multi-modal imagery and method for same
The present disclosure provides an image capturing device that captures images of a first sensor that includes a first imaging modality, a second sensor that includes a first imaging modality and a third sensor that includes a second imaging modality. A controller connected with the first sensor, the second sensor and the third sensor, wherein the controller registers an image captured by the first sensor or the second sensor to an image captured by the third sensor.
1. An image capturing device that captures images, comprising:
a first sensor configured to capture a first image of a target in a first imaging modality, wherein the first imaging modality is a depth information modality;
a second sensor configured to capture a second image of the target in a second imaging modality; and
a controller configured to:
determine a z parameter for the first image based on each pixel captured in the first image in the first imaging modality as provided by the depth information of each pixel in the first imaging modality, wherein the z parameter is determined from the depth information of each pixel in the first imaging modality that is a distance that a first imaging plane associated with the first sensor is from the target,
calibrate the first image in the first imaging modality to the second image captured by the second sensor in the second imaging modality based on the z parameter, and
register each pixel included in the first image in the first imaging modality to each corresponding pixel in the second image in the second imaging modality based on the calibration of the first image in the first imaging modality to the second image in the second imaging modality, wherein the image registration is based on the depth information calculated from the depth information modality of the first imaging modality.
2. The image capturing device of claim 1 , wherein the first imaging modality comprises color imaging.
3. The image capturing device of claim 1 , wherein the first imaging modality comprises near-infrared imaging.
4. The image capturing device of claim 1 , wherein the second imaging modality comprises thermal imaging.
5. The image capturing device of claim 1 , further comprising a light source to illuminate a subject captured by at least one of the first sensor and the second sensor.
6. The image capturing device of claim 5 , wherein the light source further comprises at least one light emitting diode.
7. The image capturing device of claim 1 , further comprising at least one actuator connected with at least one of the first sensor and the second sensor for adjusting a position of the at least one of the first sensor and the second sensor.
8. The image capturing device of claim 1 , further comprising an unmanned aerial vehicle.
9. The image capturing device of claim 1 , wherein the first sensor and the second sensor provide information to control a vehicle.
10. The image capturing device of claim 1 , further comprising a display to display an output of the registered image.
11. The image capturing device of claim 1 , wherein the first sensor and the second sensor are housed in at least one of a robot, a motor vehicle, a toy, a baby monitor and a head-mounted display.
12. The image capturing device of claim 1 , further comprising a commutation interface to remotely communicate data from the first sensor and the second sensor.
13. An image capturing device for use with a device having a controller and a communication interface, comprising:
a first sensor configured to capture a first image of a target in a first imaging modality, wherein the first imaging modality is a depth information modality;
a second sensor configured to capture a second image of the target in a second imaging modality;
a controller configured to:
determine a z parameter for the first image based on each pixel captured in the first image in the first imaging modality as provided by the depth information of each pixel in the first imaging modality, wherein the z parameter is determined from the depth information of each pixel in the first imaging modality that is a distance that a first imaging plane associated with the first sensor is from the target,
calibrate the first image in the first imaging modality to the second image captured by the second sensor in the second imaging modality based on the z parameter, and
register each pixel included in the first image in the first imaging modality to each corresponding pixel in the second image in the second imaging modality based on the calibration of the first image in the first imaging modality to the second image in the second imaging modality, wherein the image registration is based on the depth information calculated from the depth information modality of the first imaging modality; and
a communication interface configured to send the first image captured by the first sensor and the second image captured by the second sensor, to the controller.
14. The image capturing device of claim 13 , wherein the first imaging modality comprises color imaging.
15. The image capturing device of claim 13 , wherein the first imaging modality comprises near-infrared imaging.
16. The image capturing device of claim 13 , wherein the second imaging modality comprises thermal imaging.
17. The image capturing device of claim 1 , wherein the controller is further configured to:
determine the z parameter for the first image based on each pixel captured in the first image in a LIDAR imaging modality thereby providing the depth information of each pixel in the LIDAR imaging modality, wherein the z parameter is determined from the depth information of each pixel in the first imaging modality which is a distance that the first imaging plane associated with the first sensor is from the target;
calibrate the first image in the LIDAR imaging modality to the second image captured by the second sensor in the second imaging modality based on the z parameter; and
register each pixel included in the first image in the LIDAR imaging modality to each corresponding pixel in the second image in the second imaging modality based on the calibration of the first image in the LIDAR imaging modality to the second image in the second imaging modality, wherein the image registration is based on the depth information calculated from the LIDAR imaging modality.
18. The image capturing device of claim 17 , wherein the controller is further configured to:
register each pixel included in the first image in the LIDAR imaging modality to each corresponding pixel in the second image in the second imaging modality to correlate LIDAR data captured from the first image in the LIDAR imaging modality to the second image captured by the second sensor in the second imaging modality thereby enabling the second image sensor to capture a correlated image incorporating the LIDAR data into the second image captured in the second imaging modality.
19. The image capturing device of claim 13 , wherein the controller is further configured to:
determine the z parameter for the first image based on each pixel captured in the first image in a LIDAR imaging modality thereby providing the depth information of each pixel in the LIDAR imaging modality, wherein the z parameter is determined from the depth information of each pixel in the first imaging modality which is a distance that the first imaging plane associated with the first sensor is from the target;
calibrate the first image in the LIDAR imaging modality to the second image captured by the second sensor in the second imaging modality based on the z parameter; and
register each pixel included in the first image in the LIDAR imaging modality to each corresponding pixel in the second image in the second imaging modality based on the calibration of the first image in the LIDAR imaging modality to the second image in the second imaging modality, wherein the image registration is based on the depth information calculated from the LIDAR imaging modality.
20. The image capturing device of claim 19 , wherein the controller is further configured to:
register each pixel included in the first image in the LIDAR imaging modality to each corresponding pixel in the second image in the second imaging modality to correlate LIDAR data captured from the first image in the LIDAR imaging modality to the second image captured by the second sensor in the second imaging modality thereby enabling the second image sensor to capture a correlated image incorporating the LIDAR data into the second image captured in the second imaging modality.