Method and system for adaptive perspective correction of ultra wide-angle lens images
View Patent ↗Methods for correcting an image acquired through an Ultra Wide-Angle (UWA) lens is disclosed, where the entire image is mapped via a transformation that substantially improves the perspective, yet maintains all image information. Additional locally adaptive transformations can be applied to regions of interest in the image to fully restore the perspective. This de-warping scheme allows for flexible manipulation of the image while no content would be lost.
1. A method for transforming a plurality of input video images captured from a scene through at least one ultra wide-angle (UWA) lens using at least one electronic processor, wherein the at least one UWA lens is characterized by a corresponding lens transformation that maps three dimensional (3D) object space onto a plane that displays the plurality of input video images, said method comprising:
(a) obtaining the plurality of input video images data captured through the at least one UWA lens;
(b) constructing a two dimensional (2D) surface envelope in accordance with one or more areas of the scene selected for perspective correction, wherein the 2D surface envelope fully encompasses field of view of the at least one UWA lens;
(c) constructing an envelope transformation, independently from the corresponding UWA lens transformation, that maps an output image plane onto the 2D surface envelope uniquely for every point in the output image plane;
(d) concatenating the corresponding UWA lens transformation for the at least one UWA lens and the envelope transformation to obtain a zero-content-loss transformation, that provides a content preserving mapping between the input and output image planes, and wherein the UWA lens transformation is independent of image content; and
(e) applying the zero-content-loss transformation to the plurality of input video images data to obtain a plurality of output video images, wherein image perspective is accordingly transformed for the one or more selected areas of the scene;
wherein at least one of the surface envelope and the envelope transformation is modified to locally adjust for perspective corrections in at least one region of interest; and wherein the envelope transformation is modifiable without change to the surface envelope.
2. The method of claim 1 , wherein the at least one region of interest is selected manually.
3. The method of claim 1 , wherein the at least one region of interest is selected automatically in response to changes in the scene.
4. The method of claim 1 , wherein at least one of scaling, zoom, rotation and reflection transformation is applied to the at least one perspective corrected region of interest.
5. The method of claim 4 , wherein the at least one corrected region of interest is supplied for video content analysis.
6. The method of claim 1 , wherein the local perspective correction at a region of interest in a direction from the lens is obtained by:
(i) constructing a plane envelope perpendicular to an axis connecting the lens to center of the region of interest;
(ii) smoothly stitching the plane envelope and the surface envelope, and their transformations in the region of interest.
7. The method of claim 6 , wherein local magnification is achieved by moving the plane envelope along the axis connecting the lens to the region of interest.
8. The method of claim 1 , wherein the at least one UWA lens covers up to a 360° field of view.
9. The method of claim 1 , wherein the lens map is provided by the lens manufacturer.
10. The method of claim 1 , wherein the lens map is mathematically modeled.
11. The method of claim 1 , wherein the lens map is empirically characterized.
12. The method of claim 1 , wherein the surface envelope and the envelope transformation are constructed in a parameterized form.
13. The method of claim 1 , wherein a cylindrical projection is used to map the output image plane to the surface envelope.
14. The method of claim 1 , wherein mapping the output image plane to the surface envelope is represented as a function.
15. The method of claim 1 , wherein the plurality of input video images is circular and the envelope is selected as a partial surface of a sphere, having a boundary at the field of view of the lens.
16. The method of claim 1 , wherein the plurality of input video images is non-circular and the envelope is selected as a partial surface of an ellipsoid, having a boundary at a maximum field of view of the lens.
17. The method of claim 15 , wherein the envelope transformation maps an output rectangular image comprising a boundary and a plurality of pixel rows, said transformation is constructed by:
(i) mapping pixels on the output image boundary to the boundary of the ellipsoid envelope;
(ii) mapping centers of the plurality of pixel rows to an ellipse on the ellipsoid; and
(iii) mapping non-boundary pixels for each of the plurality of pixel rows to a curve on the ellipsoid, said curve constructed by interpolating a plane containing map of the center pixel and maps of corresponding two boundary pixels.
18. The method of claim 1 , wherein the zero-content-loss output image is displayed in its entirety.
19. The method of claim 1 , wherein a panoramic view is extracted from the zero-content-loss output image for display.
20. The method of claim 1 , wherein a plurality of perspective corrected regions is extracted from the output image to display independently.
21. The method of claim 20 , wherein at least one of scaling, zoom, rotation and reflection transformation is applied to the perspective corrected regions.
22. An electronic processor for transforming a plurality of input video images captured from a scene through at least one ultra wide-angle (UWA) lens, wherein each of the at least one UWA lens is characterized by a corresponding lens transformation that maps three dimensional (3D) object space onto a plane that displays the plurality of input video images, said processor comprising:
(a) means to obtain the plurality of input video images data captured through the at least one UWA lens;
(b) means to select a two dimensional (2D) surface envelope in accordance with one or more areas of the scene selected for perspective correction, wherein the 2D surface envelope fully encompasses field of view of the at least one UWA lens;
(c) means to construct an envelope transformation, independently from the corresponding UWA lens transformation, that maps an output image plane onto the 2D surface envelope uniquely for every point in the output image plane;
(d) means to concatenate the corresponding UWA lens transformation for the at least one UWA lens and the envelope transformation to obtain a zero-content-loss transformation that provides a content preserving mapping between the input and output image planes, and wherein the UWA lens transformation is independent of image content; and
(e) means to apply the zero-content-loss transformation to the plurality of input video images data to obtain a plurality of output video images, wherein image perspective is accordingly transformed for the one or more selected areas of the scene;
wherein said processor further comprises means to modify at least one of the surface envelope and the envelope transformation to locally adjust for perspective corrections in at least one region of interest; and wherein the envelope transformation is modifiable without change to the surface envelope.
23. The processor of claim 22 , wherein said processor further comprises means to enable manual selection of the at least one region of interest.
24. The processor of claim 22 , wherein said processor further comprises means to enable automatic selection of the at least one region of interest in response to changes in the scene.
25. The processor of claim 22 , wherein said processor further comprises means to apply at least one of scaling, zoom, rotation and reflection transformation to the at least one perspective corrected region of interest.