Distortion correction via analytical projection
A method for processing a stream of input images is provided. The method includes receiving a stream of input images, and applying a digital effect to the stream of input images. The digital effect is one or more from the group of: a pan, a tilt, or a zoom, of the stream of input images. The method further includes selecting an analytical projection type, from a plurality of analytical projection types, that maps pixels of the input stream of images to projected pixels of a modified stream of images, generating the modified stream of images, using the selected analytical projection type, thereby correcting a geometric distortion within the stream of input images, while applying the digital effect, and displaying the modified stream of images.
1 . A method for processing a stream of input images, the method comprising:
receiving, from a camera, a stream of input images at a first computing device during a videoconference with a remote second computing device;
selecting an analytical projection type, from a plurality of analytical projection types, that maps pixels of the input stream of images to projected pixels of a modified stream of images;
generating the modified stream of images, using the selected analytical projection type, thereby correcting a geometric distortion within the stream of input images;
applying, to the modified stream of images, a digital effect comprising a change to an elevation angle of the camera relative to a scene geometry, thereby generating an output stream of images that simulates an elevation for the camera; and
causing the output stream of images to be displayed on the first computing device and the same output stream of images to be displayed on the remote second computing device during the videoconference.
2 . The method of claim 1 , wherein the digital effect is a first digital effect, wherein the analytical projection type is a first analytical projection type, and wherein the method further comprises:
applying a second digital effect;
selecting a second analytical projection type, from the plurality of analytical projection types, in real-time, based on the second digital effect; and
updating the displayed modified stream of images, based on the second analytical projection type.
3 . The method of claim 1 , wherein the second digital effect is applied to the modified stream of images.
4 . The method of claim 3 , wherein the stream of input images is received from an image sensor, and wherein a lens associated with the image sensor has a horizontal field of view greater than 120 degrees.
5 . The method of claim 4 , wherein the lens associated with capturing the stream of input images has a lens distortion, and wherein the analytical projection type is based, at least in part, on the lens distortion.
6 . The method of claim 1 , wherein the plurality of analytical projection types comprises a rectilinearly symmetric projection.
7 . The method of claim 1 , wherein the plurality of analytical projection types comprise a plurality of discrete meshes, each of the plurality of discrete meshes being configured to correct, to varying degrees, the geometric distortion within the stream of input images.
8 . The method of claim 1 , wherein applying the digital effect comprises applying a plurality of analytical functions to a grid that maps (x 0 , y 0 , z 0 ) to (x 0 ′, y 0 ′, z 0 ′), wherein (x 0 , y 0 , z 0 ) corresponds to a pixel position in the stream of input images, and (x 0 ′, y 0 ′, z 0 ′) corresponds to a pixel position in the modified stream of images.
9 . The method of claim 8 , wherein the plurality of analytical functions are based on a first angle (φ), a second angle (Θ), and a third angle (φ cam ), the first, second, and third angles each corresponding to the digital effect on which the modified stream of images is based.
10 . The method of claim 9 , wherein the plurality of analytical functions comprises:
(
Y
yz
)
=
Y
0
cos
(
(
π
180
)
φ
)
-
Z
0
sin
(
(
π
180
)
φ
)
;
(
Z
yz
)
=
Y
0
sin
(
(
π
180
)
φ
)
+
Z
0
cos
(
(
π
180
)
φ
)
;
(
Y
yz
′
)
=
Y
yz
cos
(
(
π
180
)
φ
cam
)
+
Z
yz
sin
(
(
π
180
)
φ
cam
)
;
(
Z
yz
′
)
=
-
Y
yz
sin
(
(
π
180
)
φ
cam
)
+
Z
yz
cos
(
(
π
180
)
φ
cam
)
;
(
X
0
′
)
=
X
0
cos
(
(
π
180
)
θ
)
-
Z
yz
′
sin
(
(
π
180
)
θ
)
;
(
Z
yz
_
xz
)
=
X
0
sin
(
(
π
180
)
θ
)
+
Z
yz
′
cos
(
(
π
180
)
θ
)
;
(
Y
0
′
)
=
Y
yz
′
cos
(
-
(
π
180
)
φ
cam
)
+
Z
yz
xz
sin
(
-
(
π
180
)
φ
cam
)
;
and
(
Z
0
′
)
=
-
Y
yz
′
sin
(
-
(
π
180
)
φ
cam
)
+
Z
yz
_
xz
cos
(
-
(
π
180
)
φ
cam
)
.
11 . The method of claim 10 , wherein o and O are each non-zero angles.
12 . A system for processing a stream of input images, the system comprising:
an image sensor;
a lens associated with the image sensor, the lens having a lens distortion;
a display screen associated with the image sensor;
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the system to perform a set of operation, the set of operations comprising:
obtaining, from the image sensor, a stream of input images during a videoconference with a remote second computing device;
selecting an analytical projection type that maps pixels of the input stream of images to projected pixels of a modified stream of images, the analytical projection type being based, at least in part, on the lens distortion;
applying the analytical projection type to the stream of input images, to generate the modified stream of images, thereby correcting a geometric distortion within the stream of input images;
applying, to the modified stream of images, a digital effect comprising a change to an elevation angle of the image sensor relative to a scene geometry, thereby generating an output stream of images that simulates an elevation for the image sensor; and
causing the output stream of images to be displayed on the display screen and the same output stream of images to be separately displayed on the remote second computing device during the videoconference.
13 . The system of claim 12 , wherein the digital effect is a first digital effect, wherein the analytical projection type is a first analytical projection type, and wherein the set of operations further comprises:
applying a second digital effect;
selecting a second analytical projection type, from the plurality of analytical projection types, in real-time, based on the second digital effect; and
updating the modified stream of images, based on the second analytical projection type.
14 . The system of claim 12 , wherein the second digital effect is applied to the stream of input images.
15 . The system of claim 12 , wherein the lens associated with the image sensor has a horizontal field of view greater than 120 degrees.
16 . The system of claim 12 , wherein the digital effect includes one or more from the group of: a pan, a tilt, or a zoom, of the stream of input images.
17 . The system of claim 12 , wherein the plurality of analytical projection types comprise a plurality of discrete meshes, each of the plurality of discrete meshes being configured to correct, to varying degrees, the geometric distortion within the stream of input images.
18 . The system of claim 12 , wherein applying the digital effect comprises applying a plurality of analytical functions to a grid that maps (x 0 , y 0 , z 0 ) to (x 0 ′, y 0 ′, z 0 ′), wherein (x 0 , y 0 , z 0 ) corresponds to a pixel position in the stream of input images, and (x 0 ′, y 0 ′, z 0 ′) corresponds to a pixel position in the modified stream of images, and wherein the plurality of analytical functions are based on a first angle (φ), a second angle (Θ), and a third angle (φ cam ), the first, second, and third angles each corresponding to the digital effect on which the modified stream of images is based.
19 . The system of claim 18 , wherein the plurality of analytical functions comprises:
(
Y
yz
)
=
Y
0
cos
(
(
π
180
)
φ
)
-
Z
0
sin
(
(
π
180
)
φ
)
;
(
Z
yz
)
=
Y
0
sin
(
(
π
180
)
φ
)
+
Z
0
cos
(
(
π
180
)
φ
)
;
(
Y
yz
′
)
=
Y
yz
cos
(
(
π
180
)
φ
cam
)
+
Z
yz
sin
(
(
π
180
)
φ
cam
)
;
(
Z
yz
′
)
=
-
Y
yz
sin
(
(
π
180
)
φ
cam
)
+
Z
yz
cos
(
(
π
180
)
φ
cam
)
;
(
X
0
′
)
=
X
0
cos
(
(
π
180
)
θ
)
-
Z
yz
′
sin
(
(
π
180
)
θ
)
;
(
Z
yz
_
xz
)
=
X
0
sin
(
(
π
180
)
θ
)
+
Z
yz
′
cos
(
(
π
180
)
θ
)
;
(
Y
0
′
)
=
Y
yz
′
cos
(
-
(
π
180
)
φ
cam
)
+
Z
yz
xz
sin
(
-
(
π
180
)
φ
cam
)
;
and
(
Z
0
′
)
=
-
Y
yz
′
sin
(
-
(
π
180
)
φ
cam
)
+
Z
yz
_
xz
cos
(
-
(
π
180
)
φ
cam
)
.
20 . A method for processing a stream of input images, the method comprising:
receiving, from a camera, a stream of input images at a first computing device during a video conference with a remote second computing device;
selecting an analytical projection type, from a plurality of analytical projection types, that maps pixels of the input stream of images to projected pixels of a modified stream of images;
applying the analytical projection type to the stream of input images, to generate the modified stream of images, thereby correcting a geometric distortion within the stream of input images, in the modified stream of images;
applying, to the modified stream of images, a digital effect comprising a change to an elevation angle of the camera relative to a scene geometry, thereby generating an output stream of images that simulates an elevation for the camera; and
causing the modified stream of images to be displayed on the remote second computing device during the videoconference.