Image alignment using a virtual gyroscope model
A target image captured from a fisheye lens or other lens with known distortion parameters may be transformed to align it to a reference image. Corresponding features may be detected in the target image and the reference image. The features may be transformed to a spherical coordinate space. In the spherical space, images may be re-pointed or rotated in three dimensions to align all or a subset of the features of the target image to the corresponding features of the reference image. For example, in a sequence of images, background features of the target image in the spherical image space may be aligned to background features of the reference image in the spherical image space to compensate for camera motion while preserving foreground motion. An inverse transformation may then be applied to bring the images back into the original image space.
1. A method for aligning a target image to a reference image in the presence of lens distortion, the method comprising:
receiving the target image and the reference image;
detecting a first plurality of visual features appearing in the target image at first image feature coordinates in a two-dimensional image space, and a corresponding second plurality of visual features appearing in the reference image at second image feature coordinates in the two-dimensional image space;
transforming the first image feature coordinates and the second image feature coordinates from the two-dimensional image space to a three-dimensional spherical space to generate respective first spherical feature coordinates and second spherical feature coordinates;
applying a rotation to the target image in the three-dimensional spherical space to generate a rotated target image by at least:
determining at least a subset of the first spherical feature coordinates and a corresponding subset of the second spherical feature coordinates as background features in the target image and the reference image, respectively;
determining feature tracks representing a distance along a spherical arc and direction from the first spherical feature coordinates to a corresponding one of the second spherical feature coordinates;
clustering the feature tracks by length to determine a plurality of feature track clusters; and
determining, from the plurality of feature track clusters, a background cluster of feature tracks corresponding to the background features; and
inverse transforming the rotated target image to the two-dimensional image space.
2. The method of claim 1 , wherein the determining of the background cluster comprises:
determining a predefined number of largest clusters from the plurality of feature track clusters;
determining weighted averages of track lengths in each of the largest clusters; and
selecting a cluster having a shortest weighted average of the weighted averages.
3. The method of claim 1 , wherein the applying of the rotation further comprises:
determining first three-dimensional Cartesian coordinates corresponding to the subset of the first spherical feature coordinates and second three-dimensional Cartesian coordinates corresponding to the subset of the second spherical feature coordinates;
determining a first coordinate centroid of the first three-dimensional Cartesian coordinates and a second coordinate centroid of the second three-dimensional Cartesian coordinates;
determining first translated coordinates as a difference between the first three-dimensional Cartesian coordinates and the first coordinate centroid, and second translated coordinates as a difference between the second three-dimensional Cartesian coordinates and the second coordinate centroid;
determining products between each of the first translated coordinates and respective corresponding second translated coordinates;
determining an accumulation matrix as a combination of the products; and
applying a singular value decomposition to the accumulation matrix to determine a rotation matrix, the rotation matrix indicating the rotation to apply to the target image in the three-dimensional spherical space.
4. The method claim 1 , further comprising capturing the target image and the reference image with a lens having lens distortion parameters, the lens distortion parameters corresponding to a fisheye lens that introduces curvature into straight lines in a scene near edges of a captured image.
5. The method of claim 1 , wherein the applying of the rotation further comprises optimally aligning the subset of the first spherical feature coordinates to the corresponding subset of the second spherical feature coordinates based on an optimization metric.
6. The method of claim 1 , further comprising storing the lens distortion parameters within the target image and the reference image.
7. The method of claim 1 , further comprising determining from the plurality of feature track clusters a foreground object that is distinct from the background features.
8. A non-transitory computer-readable apparatus comprising a computer program configured to align a target image to a reference image in the presence of lens distortion, the computer program comprising instructions that, when executed by a processor apparatus, cause the processor apparatus to:
receive the target image and the reference image;
detect a first plurality of visual features appearing in the target image at first image feature coordinates in a two-dimensional image space, and a corresponding second plurality of visual features appearing in the reference image at second image feature coordinates in the two-dimensional image space;
transform the first image feature coordinates and the second image feature coordinates from the two-dimensional image space to a three-dimensional spherical space to generate respective first spherical feature coordinates and second spherical feature coordinates;
apply a rotation to the target image in the three-dimensional spherical space to generate a rotated target image via at least:
determination of at least a subset of the first spherical feature coordinates and a corresponding subset of the second spherical feature coordinates as background features in the target image and the reference image, respectively;
determination of feature tracks representing a distance along a spherical arc and direction from the first spherical feature coordinates to a corresponding one of the second spherical feature coordinates;
cluster the feature tracks by length to determine a plurality of feature track clusters; and
determine, from the plurality of feature track clusters, a background cluster of feature tracks corresponding to the background features; and
inverse transform the rotated target image to the two-dimensional image space.
9. The non-transitory computer-readable apparatus of claim 8 , wherein the determination of the background cluster further comprises:
determination of a predefined number of largest clusters from the plurality of feature track clusters;
determine weighted averages of track lengths in each of the largest clusters; and
select a cluster having a shortest weighted average of the weighted averages.
10. The non-transitory computer-readable apparatus of claim 8 , wherein the application of the rotation further comprises:
a determination of first three-dimensional Cartesian coordinates corresponding to the subset of the first spherical feature coordinates and second three-dimensional Cartesian coordinates corresponding to the subset of the second spherical feature coordinates;
determine a first coordinate centroid of the first three-dimensional Cartesian coordinates and a second coordinate centroid of the second three-dimensional Cartesian coordinates;
determine first translated coordinates as a difference between the first three-dimensional Cartesian coordinates and the first coordinate centroid, and second translated coordinates as a difference between the second three-dimensional Cartesian coordinates and the second coordinate centroid;
determine products between each of the first translated coordinates and respective corresponding second translated coordinates;
determine an accumulation matrix as a combination of the products;
apply a singular value decomposition to the accumulation matrix to determine a rotation matrix, the rotation matrix indicating the rotation to apply to the target image in the three-dimensional spherical space.
11. The non-transitory computer-readable apparatus of claim 8 , wherein the target image and the reference image are captured with a lens having lens distortion parameters, the lens distortion parameters corresponding to a fisheye lens that introduces curvature into straight lines in a scene near edges of a captured image.
12. The non-transitory computer-readable apparatus of claim 8 , wherein the application of the rotation comprises optimal alignment of the subset of the first spherical feature coordinates to the corresponding subset of the second spherical feature coordinates based on an optimization metric.
13. The non-transitory computer-readable apparatus of claim 8 , wherein the determination of the background cluster comprises a determination of the plurality of feature track clusters having a cluster that has a largest size.
14. The non-transitory computer-readable apparatus of claim 8 , wherein the instructions are configured to, when executed by the processor apparatus, further apply a translation to the target image in the three-dimensional spherical space to generate the rotated target image.
15. A video server comprising:
one or more processors; and
a non-transitory computer-readable apparatus comprising a computer program configured to align a target image to a reference image in the presence of lens distortion, the computer program comprising instructions that, when executed by the one or more processors, causes the one or more processors to:
receive the target image and the reference image;
detect a first plurality of visual features appearing in the target image at first image feature coordinates in a two-dimensional image space, and a corresponding second plurality of visual features appearing in the reference image at second image feature coordinates in the two-dimensional image space;
transform the first image feature coordinates and the second image feature coordinates from the two-dimensional image space to a three-dimensional spherical space to generate respective first spherical feature coordinates and second spherical feature coordinates;
apply a rotation to the target image in the three-dimensional spherical space to generate a rotated target image via at least:
determination of at least a subset of the first spherical feature coordinates and a corresponding subset of the second spherical feature coordinates as background features in the target image and the reference image, respectively;
determination of feature tracks representing a distance along a spherical arc and direction from the first spherical feature coordinates to a corresponding one of the second spherical feature coordinates;
cluster the feature tracks by length to determine a plurality of feature track clusters; and
determine, from the plurality of feature track clusters, of a background cluster of feature tracks corresponding to the background features; and
inverse transform the rotated target image to the two-dimensional image space.
16. The video server of claim 15 , wherein the determination of the background cluster comprises:
determination of a predefined number of largest clusters from the plurality of feature track clusters;
determination of weighted averages of track lengths in each of the largest clusters; and
selection of a cluster having a shortest weighted average of the weighted averages.
17. The video server of claim 15 , wherein the application of the rotation further comprises:
determination of first three-dimensional Cartesian coordinates corresponding to the subset of the first spherical feature coordinates and second three-dimensional Cartesian coordinates corresponding to the subset of the second spherical feature coordinates;
determine a first coordinate centroid of the first three-dimensional Cartesian coordinates and a second coordinate centroid of the second three-dimensional Cartesian coordinates;
determine first translated coordinates as a difference between the first three-dimensional Cartesian coordinates and the first coordinate centroid and second translated coordinates as a difference between the second three-dimensional Cartesian coordinates and the second coordinate centroid;
determine products between each of the first translated coordinates and respective corresponding second translated coordinates;
determine an accumulation matrix as a combination of the products; and
application of a singular value decomposition to the accumulation matrix to determine a rotation matrix, the rotation matrix being configured to indicate the rotation to apply to the target image in the three-dimensional spherical space.
18. The video server of claim 15 , wherein the application of the rotation further comprises an alignment of the subset of the first spherical feature coordinates to the corresponding subset of the second spherical feature coordinates based on an optimization metric.
19. The video server of claim 15 , wherein the video server is configured to be in data communication with one or more of a camera device and a client device via a network.
20. The video server of claim 15 , wherein the instructions are further configured to, when executed by the one or more processors, cause the one or more processors to extract lens distortion parameters from one or more of the target image and the reference image.