Stereoscopic 360-degree image/video generation and 3D display system
View Patent ↗A stereoscopic 360-degree image/video generation and 3D display system includes a generation platform configured to generate a stereoscopic 360-degree image/video by inferring a disparity map from a 360-degree image/video, and a display platform configured to perform three-dimensional (3D) display on a 3D display device by scanning the stereoscopic 360-degree image/video generated by the generation platform. The stereoscopic 360-degree image/video generation 3D display system is divided into a stereoscopic 360-degree image/video generation platform that requires high throughput and a platform that three-dimensionally displays a low-latency stereoscopic 360-degree image/video, and thus has an effect of being able to efficiently achieve a platform configuration and hardware design for satisfying requirements in each platform.
1 . A stereoscopic 360-degree image/video generation and 3D display system comprising:
a generation platform configured to generate a stereoscopic 360-degree image/video by inferring a disparity map from a 360-degree image/video; and
a display platform configured to perform three-dimensional (3D) display on a 3D display device by scanning the stereoscopic 360-degree image/video generated by the generation platform,
wherein the generation platform comprises:
an image capturing module configured to generate a 360-degree image by utilizing multiple cameras or a time-based image overlay technique;
a disparity map generation module configured to project images corresponding to tangent surfaces of spherical coordinate systems onto the 360-degree image, then infer a disparity map of each image using artificial intelligence (AI) technology, and inverse-project this disparity map onto the spherical coordinate system to generate a 360-degree image disparity map; and
an opposite eye viewpoint image generation module configured to generate a 360-degree image corresponding to another field of view by utilizing the 360-degree image and the 360-degree image disparity map.
2 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 1 , wherein the disparity map generation module generates a 360-degree image disparity map by directly applying AI technology to the 360-degree image.
3 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 1 , wherein the opposite eye viewpoint image generation module utilizes the 360-degree image and the 360-degree image disparity map to warp an original image pixel to an image coordinate system corresponding to an opposite eye according to a disparity value in the 360-degree image corresponding to an original in an equirectangular image coordinate system, thereby generating a 360-degree image corresponding to a viewpoint of the opposite eye.
4 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 1 , wherein the opposite eye viewpoint image generation module projects the 360-degree image and a 360-degree disparity map in several viewpoint directions sufficient to encompass all spherical coordinate system information corresponding to tangent surfaces of spherical coordinate systems, then performs warping to an image coordinate system corresponding to an opposite eye using each image and a disparity map corresponding thereto, and inverse-projects again and then combines results thereof, thereby generating a 360-degree image of the opposite eye.
5 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 3 , wherein the opposite eye viewpoint image generation module fills a missing part of an untilled image with image pixels using inpainting technology in the generated 360-degree image of the opposite eye.
6 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 4 , wherein the opposite eye viewpoint image generation module fills a missing part of an untilled image with image pixels using inpainting technology in the generated 360-degree image of the opposite eye.
7 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 1 , wherein the display platform comprises:
an image loading module configured to load a 360-degree image generated by each of the image capturing module and the opposite eye viewpoint image generation module;
a tangent surface image projection module configured to project 360-degree images of two eyes loaded by the image loading module onto tangent surfaces through image projection in accordance with target viewing angles, thereby generating tangent surface images at positions in a field of view of a person; and
a display module configured to scan two tangent surface images at the positions in the field of view of the person received from the tangent surface image projection module on a screen corresponding to the positions of the two eyes of the person.
8 . The stereoscopic 360-degree image/video generation and 3D display system according to claim 7 , wherein, when the 360-degree images of the two eyes are projected onto the tangent surface images of the two eyes, the tangent surface image projection module performs calculation through [Mathematical Formula 1] or [Mathematical Formula 2] below, and projects an image projection result of a viewpoint of one eye based on [Mathematical Formula 1] or [Mathematical Formula 2] below onto an image of a viewpoint of the other eye to generate the tangent surface images of the two eyes from the 360-degree images of the two eyes:
[
Mathematical
Formula
1
]
u
=
cos
(
Φ
)
sin
(
θ
-
θ
c
)
cos
(
c
)
,
v
=
cos
(
Φ
c
)
sin
(
Φ
)
-
sin
(
Φ
c
)
cos
(
Φ
)
cos
(
Φ
-
Φ
c
)
cos
(
c
)
,
cos
(
c
)
=
sin
(
Φ
c
)
sin
(
Φ
)
+
cos
(
Φ
c
)
cos
(
Φ
)
cos
(
θ
-
θ
c
)
,
and
[
Mathematical
Formula
2
]
θ
-
θ
c
+
tna
-
1
(
u
·
sin
(
c
)
r
cos
(
Φ
c
)
cos
(
c
)
-
v
·
sin
(
Φ
c
)
sin
(
c
)
)
,
Φ
=
sin
-
1
(
cos
(
c
)
sin
(
Φ
c
)
+
1
r
v
·
sin
(
c
)
cos
(
Φ
c
)
)
r
=
u
2
+
v
2
and
c
=
tan
-
1
r
where θ denotes a longitude in a spherical coordinate system, φ denotes a latitude in the spherical coordinate system, θ c denotes a longitude in a spherical coordinate system corresponding to a center of a tangent surface image, Qc denotes a latitude in the spherical coordinate system corresponding to the center of the tangent surface image, u denotes a horizontal coordinate in the tangent surface image, and v denotes a vertical coordinate in the tangent surface image.