2D-to-3D video frame conversion
A wide spread adoption of 3D videos and technologies is hindered by the lack of high-quality 3D content. One promising solution to address this problem is to use automated 2D-to-3D conversion. However, current conversion methods, while general, produce low-quality results with artefacts that are not acceptable to many viewers. Creating a database of 3D stereoscopic videos with accurate depth is, however, very difficult. Computer generated content can be used to generate high-quality 3D video reference database for 2D-to-3D conversion. The method transfers depth information from frames in the 3D reference database to the target frame while respecting object boundaries. It computes depth maps from the depth gradients, and outputs a stereoscopic video.
1. A method of processing 2D video images from a video stream for converting the 2D video images to 3D images, the method comprising:
providing a synthetic 3D reference database of video frames, each entry in the database comprising a 2D image and corresponding depth information for that 2D image;
submitting input video frames to the synthetic 3D reference database;
dividing the input video frame into portions;
matching a portion of the input video frame with a portion of a 2D image in the synthetic 3D reference database;
and selecting the corresponding depth information for that 2D image, wherein the depth information is a depth gradient; and
applying the selected depth information to the matched input video frame portion to generate a 2D plus depth information frame portion.
2. The method of claim 1 wherein the portions are blocks of n×n pixels.
3. The method of claim 1 , wherein the method further comprises matching another portion of the input video frame with a portion of the 2D image or another 2D image in the synthetic 3D reference database.
4. The method of claim 1 , wherein applying the selected depth information to the matched input video frame comprises applying the depth information of the matched portion of the 2D image to the respective matched portion of the matched input video frame.
5. The method of claim 4 wherein applying the selected depth information to the matched input video frame comprises mapping one or more corresponding pixels of the matched portion of the 2D image to the corresponding pixels of the matched portion of the input video frame.
6. The method of claim 1 , comprising:
identifying a candidate 2D image for matching with the input video frame.
7. The method of claim 6 wherein said identifying comprises using GIST and colour information of the frames.
8. The method of claim 1 , further comprising:
identifying objects in the input video frame;
determining object masks for the identified objects; and
estimating the depth information using the determined object masks and the matched input video frame, allowing depth discontinuities at object boundaries by modifying the Poisson equation there.
9. The method of claim 8 further comprising:
estimating the depth information using a Poisson reconstruction, formulated as Ax=b, where b=∇G, x=D, and A stores the coefficients of the Poisson equation, comprising:
disconnecting pixels from object boundaries by not allowing them to use an object boundary pixel as a valid neighbour, and for each pixel neighbouring a boundary pixel, setting the corresponding connection in A to 0 and updating its ∇ G value accordingly so that pixels adjacent to object boundaries are treated similar to image boundary pixels.
10. The method of claim 9 , wherein the Poisson reconstruction comprises first order and higher derivatives.
11. The method of claim 1 , further comprising generating a left stereo image and a right stereo image using the 2D plus depth information frame.
12. The method of claim 1 , wherein the reference database is populated using software generated depth information for 2D images.
13. The method of claim 12 , wherein the software is a video game.
14. A system to process 2D video images from a video stream for converting the 2D video images to 3D images, the system comprising:
a synthetic 3D reference database of video frames, each entry in the database comprising a 2D image and corresponding depth information for that 2D image;
a search module configured to submit input video frames to the synthetic 3D reference database;
a matching module configured to match a portion of an input video frame with a portion of a 2D image in the synthetic 3D reference database, the system being configured to:
divide the input video frame into portions and match portion of the input video frame with a portion of a 2D image in the synthetic 3D reference database; and
selecting the corresponding depth information for that 2D image, wherein the depth information is a depth gradient;
the system further comprising:
a generator module configured to apply the selected depth information to the matched input video frame portion to generate a 2D plus depth information frame portion.
15. A non-transitory computer-readable medium programmed with instructions that when executed, perform the method of claim 1 .
16. The method of claim 1 , comprising:
reconstructing depth information using a Poisson reconstruction, according to:
(
∂
2
∂
x
2
+
∂
2
∂
y
2
)
D
=
∇
·
G
,
where G=(G x ,G y ) is the depth gradient, D is the depth information and ∇G is the divergence of G:
∇
·
G
=
(
∂
G
x
∂
x
+
∂
G
y
∂
y
)
.
17. The method of claim 16 , further comprising refining depth gradients using:
G
x
=
G
x
×
max
(
1
-
e
(
1
-
1
α
G
x
)
,
0
)
G
y
=
G
y
×
max
(
1
-
e
(
1
-
1
α
G
y
)
,
0
)
.
18. The method of claim 16 , comprising forcing the higher-order depth derivatives to be zero, comprising setting, in the continuous domain:
(
∂
4
∂
x
4
+
∂
4
∂
y
4
)
D
=
0.