Granted Patent B2

US 12,548,237 · App. 18/393,078 · Granted Feb 10, 2026

Inventors: Junghyun Cho (Seoul, KR); Ig Jae Kim (Seoul, KR); Haesol Park (Seoul, KR); Jun Yong Choi (Seoul, KR); Seok Yeong Lee (Seoul, KR)

Assignee: Korea Institute of Science and Technology

G06T15/20G06T15/506

Abstract

Claims (38)

1 . An apparatus for synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, the apparatus comprising:

a target view analysis unit configured to estimate a normal map and direct lighting using a plurality of 2D images;

a material estimation unit configured to reflect the normal map and direct lighting estimated from the target view analysis unit and estimate material information;

a 3D lighting estimation unit configured to reflect the direct lighting estimated from the target view analysis unit and the material information estimated from the material estimation unit and estimate 3D spatially varying lighting information; and

an image generation unit configured to reflect the estimated shapes, materials, and 3D spatially varying lighting information, synthesize the 2D images with 3D objects, and generate a synthesized image.

2 . The apparatus of claim 1 , wherein the target view analysis unit includes:

a normal map estimation unit configured to receive RGB images acquired from the plurality of 2D images, a depth map, and a confidence map and estimate a normal map from the depth map;

an incident lighting estimation unit configured to estimate incident direct lighting using the RGB images, the depth map, the confidence map, and the normal map; and

an exitant lighting estimation unit configured to estimate exitant direct lighting using the RGB images, the depth map, the confidence map, and the normal map.

3 . The apparatus of claim 1 , wherein the material estimation unit includes:

a specular radiance estimation unit configured to estimate a specular radiance feature (f spec ) based on the normal map and an incident direct lighting estimated value (η, λ, ξ) estimated from the target view analysis unit;

a context estimation unit configured to estimate a context feature (f context ) based on the RGB images, the depth map, and the confidence map;

a multi-view aggregation unit configured to aggregate the specular radiance feature (f spec ), the context feature (f context ), and multi-view RGB images and generate a multi-view aggregation feature; and

an albedo and roughness estimation unit configured to estimate albedo and roughness based on the RGB images, the depth map, the confidence map, the context feature (f context ), and a bidirectional reflectance distribution function (BRDF) feature (f BRDF ) output from the multi-view aggregation unit.

4 . The apparatus of claim 1 , wherein the 3D lighting estimation unit reflects the RGB images, the depth map, the confidence map, an exitant direct lighting volume estimated from the target view analysis unit, and the albedo and roughness information estimated from the material estimation unit, estimates the 3D spatially varying lighting information, and outputs the 3D spatially varying lighting volume.

5 . The apparatus of claim 1 , wherein the image generation unit reflects the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimation unit and generates a novel view image.

6 . The apparatus of claim 1 , wherein the image generation unit reflects the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimation unit, synthesizes sounds or 3D objects and the sounds into the 2D images, and generates a synthesized image.

7 . A method of synthesizing a 2D image and media using a reverse rendering including 3D spatially varying lighting information estimation, the method comprising:

a target view analyzing operation of estimating a normal map and direct lighting using a plurality of 2D images;

a material estimating operation of reflecting the estimated normal map and direct lighting and estimating material information;

a 3D lighting estimating operation of reflecting the estimated direct lighting and material information and estimating 3D spatially varying lighting information; and

an image generating operation of reflecting the estimated shapes, materials, and 3D spatially varying lighting information, synthesizing the 2D images with 3D objects, and generating an image.

8 . The method of claim 7 , wherein the target view analyzing operation includes:

receiving RGB images acquired from the plurality of 2D images, a depth map, and a confidence map;

estimating a normal map from the depth map;

estimating incident direct lighting using the RGB images, the depth map, the confidence map, and the normal map; and

estimating exitant direct lighting using the RGB images, the depth map, the confidence map, and the normal map.

9 . The method of claim 7 , wherein the material estimating operation includes:

estimating a specular radiance feature (f spec ) based on the normal map and an incident direct lighting estimated value (η, λ, ξ) estimated from the target view analyzing operation;

estimating a context feature (f context ) based on the RGB images, the depth map, and the confidence map;

aggregating the specular radiance feature (f spec ), the context feature (f context ), and multi-view RGB images and generating a multi-view aggregation feature; and

estimating albedo and roughness based on the RGB images, the depth map, the confidence map, the context feature (f context ), and a bidirectional reflectance distribution function (BRDF) feature (f BRDF ) output from a multi-view aggregation unit configured to generate the multi-view aggregation feature.

10 . The method of claim 7 , wherein the 3D lighting estimating operation includes reflecting the RGB images, the depth map, the confidence map, an exitant direct lighting volume estimated from the target view analyzing operation, and the albedo and roughness information estimated from the material estimating operation, estimating the 3D spatially varying lighting information, and outputting the 3D spatially varying lighting volume.

11 . The method of claim 7 , wherein the image generating operation includes:

setting positions of the 3D objects to be synthesized into a 2D image when the target 2D image and the 3D objects are input; and

calculating lighting information at the set positions of the 3D objects using the 3D spatially varying lighting information volume acquired through the 3D lighting estimating operation and generating an image by rendering the 3D objects including shadows based on the calculated lighting information.

12 . The method of claim 11 , wherein the image generating operation further includes generating a novel view image using the 3D spatially varying lighting volume acquired through the 3D lighting estimating operation.

13 . The method of claim 7 , wherein the image generating operation includes reflecting the shapes, materials, and 3D spatially varying lighting information estimated from the 3D lighting estimating operation, synthesizing sounds or 3D objects and the sounds into the 2D images, and generating a synthesized image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (15)