Motion compensated image registration for overlaid/fused video
View Patent ↗A system for compensating image misregistration between at least two image sensors includes a first image sensor and a second image sensor which are disposed on a platform and are configured to provide first and second output images, respectively. The system also includes a motion sensor and a processor. The motion sensor senses movement of the platform. The processor calculates a lag time between the first and second image sensors based on first and second processing delay times of the first and second image sensors, respectively. The processor also calculates an image offset based on the lag time and the movement of the platform sensed by the motion sensor and offsets one of the first or second output image with respect to the other one of the first or second output image based on the image offset. A fuser combines the offset image with the non-offset image.
1. A system for compensating image misregistration between at least two image sensors comprising:
first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;
a motion sensor for sensing movement of the platform; and
a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;
wherein the image offset is calculated by determining (a) horizontal and vertical velocities of the platform, and (b) processing delay time between the first and second output images, and
the processor provides a vector defining a number of pixels in horizontal rows and vertical columns by which the first and second output images are shifted to compensate for the image misregistration between the first and second output images;
wherein the processor is configured to calculate horizontal and vertical image offsets using the following equation:
offset x,y =υ x,y Lag T ,
where:
υ x,y is a velocity vector of the horizontal and vertical linear velocities,
Lag T is the lag time, and
offset x,y is a vector of the calculated horizontal and vertical image offsets.
2. The system according to claim 1 including an image fuser,
wherein the processor is configured to (i) calculate the image offset further based on the lag time and (ii) offset one of the first or second output image with respect to the other one of the first or second output image based on the image offset, and
the fuser is configured to fuse the first and second output images based on the image offset.
3. The system according to claim 1 , wherein the processor is configured to calculate horizontal and vertical linear velocities in a plane of the first or second output image based on the sensed movement.
4. The system according to claim 1 including an image fuser configured to fuse the first and second output images, after offsetting the first or second output image.
5. The system according to claim 4 , wherein a synchronization signal is provided to the image fuser as a time reference between the first and second output images.
6. The system according to claim 1 , wherein the processor is configured to calculate a vertical linear velocity of the first or second output image, using the following equation:
υ
y
=
ω
v
Rows
FOV
v
,
where: ω v is a vertical angular velocity of the platform,
Rows is a number of pixels in a column of the first or second output image,
FOV v is a vertical field of view of the first or second image sensor, and
υ y is the calculated vertical linear velocity of the first or second output image.
7. A system for compensating image misregistration between at least two image sensors comprising:
first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;
a motion sensor for sensing movement of the platform;
a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;
the motion sensor includes an accelerometer configured to sense horizontal and vertical linear accelerations of the platform; and
an integrator for integrating over time the horizontal and vertical linear accelerations to provide respective horizontal and vertical linear velocities of the platform; and
first and second frame memories configured to store respective first and second previous frames of the first or second output image, and wherein the processor is configured to zero the integrator when the processor determines that the first and second previous frames are spatially correlated to within an error limit.
8. The system according to claim 7 wherein the processor is configured to calculate horizontal and vertical linear velocities of the first or second output image based on the horizontal and vertical linear velocities of the platform.
9. The system according to claim 7 , wherein the processor is configured to calculate the horizontal linear velocity of the first or second output image using the following equation:
υ
x
=
(
υ
h
r
Cols
)
FOV
h
,
where:
υ h is the horizontal linear velocity of the platform,
r is a radial distance from the accelerometer to a pivot point of the platform,
Cols is a number of pixels in a row of the first or second output image,
FOV h is a horizontal field of view of the first or second image sensor, and
υ x is the horizontal linear velocity of the first or second output image.
10. The system according to claim 7 , wherein the processor is configured to calculate the vertical linear velocity of the first or second output image using the following equation:
υ
y
=
(
υ
v
r
Rows
)
FOV
v
,
where:
υ v is the vertical linear velocity of the platform,
r is a radial distance from the accelerometer to a pivot point of the platform,
Rows is a number of pixels in a column of the first or second output image,
FOV v is a vertical field of view of the first or second image sensor, and
υ y is the vertical linear velocity of the first or second output image.
11. The system according to claim 7 , wherein the first image sensor includes an optical system configured to provide the first output image, and
the second image sensor includes an electronic image sensor configured to provide the second output image.
12. The system according to claim 11 including an image fuser comprising a digital display and a combining prism, wherein:
the optical system is configured to project onto the combining prism the first output image,
the digital display is configured to project onto the combining prism a light image based upon the second output image, the image offset, and the lag time, and
the combining prism is configured to fuse the first image with the projected light image.
13. A system for compensating image misregistration between at least two image sensors comprising:
first and second image sensors disposed on a platform and configured to provide first and second output images, respectively;
a motion sensor for sensing movement of the platform; and
a processor configured to calculate an image offset based on the movement of the platform sensed by the motion sensor;
wherein the motion sensor includes a rate sensor configured to sense angular velocities of the platform, and
the processor is configured to calculate a horizontal linear velocity of the first or second output image, using the following equation:
υ
x
=
ω
h
Cols
FOV
h
,
where:
ω h is a horizontal angular velocity of the platform,
Cols is a number of pixels in a row of the first or second output image,
FOV h is a horizontal field of view of the first or second image sensor, and
υ x is the calculated horizontal linear velocity of the first or second output image.
14. The system according to claim 13 , wherein the first and second image sensors include, respectively, first and second processing delay times, and
the processor is configured to calculate a lag time between the first and second image sensors based on the first and second processing delay times.
15. The system according to claim 14 including an image fuser,
wherein the processor is configured to (i) calculate the image offset further based on the lag time and (ii) offset one of the first or second output image with respect to the other one of the first or second output image based on the image offset, and
the fuser is configured to fuse the first and second output images based on the image offset.
16. The system according to claim 14 , wherein the processor is configured to calculate horizontal and vertical linear velocities in a plane of the first or second output image based on the sensed movement.
17. The system according to claim 16 , wherein the processor is configured to offset the first or second output image based on the calculated horizontal and vertical linear velocities.
18. The system according to claim 17 including an image fuser configured to fuse the first and second output images, after offsetting the first or second output image.
19. The system according to claim 18 , wherein a synchronization signal is provided to the image fuser as a time reference between the first and second output images.
20. The system according to claim 13 , wherein the processor is configured to calculate a vertical linear velocity of the first or second output image, using the following equation:
υ
y
=
ω
v
Rows
FOV
v
,
where:
ω v is a vertical angular velocity of the platform,
Rows is a number of pixels in a column of the first or second output image,
FOV v is a vertical field of view of the first or second image sensor, and
υ y is the calculated vertical linear velocity of the first or second output image.
21. The system according to claim 13 , wherein the processor is configured to calculate horizontal and vertical image offsets using the following equation:
offset x,y =υ x,y Lag T ,
where:
υ x,y is a velocity vector of the horizontal and vertical linear velocities,
Lag T is the lag time, and
offset x,y is a vector of the calculated horizontal and vertical image offsets.