Granted Patent B2

US 12,249,026 · App. 17/699,629 · Granted Mar 11, 2025

Inventor: Inwoo Ha (Seongnam-si, KR)

Assignee: Samsung Electronics Co., Ltd.

G06T15/80G06T7/70G06T15/60G06T2207/20081

Abstract

Claims (85)

1. A processor-implemented method for light estimation, comprising:

estimating light information corresponding to an input image using a light estimation model;

detecting a reference object in the input image;

determining object information of the reference object and plane information of a reference plane supporting the reference object;

rendering, based on the light information, the object information, and the plane information, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

training the light estimation model by updating the light estimation model based on a result of comparing the reference object and the rendered virtual object,

wherein the determining comprises, in response to the reference object and the reference plane being determined as a known combined structure, determining at least a portion of the object information and the plane information using an object database.

2. A processor-implemented method for light estimation, comprising:

estimating light information corresponding to an input image using a light estimation model;

detecting a reference object in the input image;

determining object information of the reference object and plane information of a reference plane supporting the reference object;

rendering, based on the light information, the object information, and the plane information, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

training the light estimation model by updating the light estimation model based on a result of comparing the reference object and the rendered virtual object,

wherein the result of comparing comprises a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data of the rendered shading of the virtual object and the rendered shadow of the virtual object.

3. The method of claim 2 , wherein

the object information comprises any one or any combination of any two or more of a pose, a shape, and a material of the reference object, and

the plane information comprises any one or any combination of any two or more of a pose, a shape, and a material of the reference plane.

4. The method of claim 2 , wherein the determining comprises, in response to the reference object being determined as a known object, determining at least a portion of the object information using an object database.

5. The method of claim 2 , wherein the determining comprises, in response to the reference plane being determined as an unknown plane, determining the plane information by detecting the reference plane in the input image.

6. The method of claim 2 , wherein the rendering comprises:

determining shadow information of each sampling point of the reference plane by fusing light information and visibility information of each sampling point; and

performing the rendering of the shadow of the virtual object by projecting the shadow information of each sampling point of the reference plane onto a capture view of the input image.

7. The method of claim 6 , wherein, in response to the reference object and the reference plane being determined as a known combined structure, the visibility information is predetermined for rendering a shadow of the combined structure.

8. The method of claim 2 , wherein the updating comprises updating the light estimation model to reduce a difference between the reference object and the virtual object.

9. The method of claim 2 , further comprising:

obtaining another image; and

estimating light information corresponding to the other image using the trained light estimation model.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 2 .

11. The method of claim 2 , wherein the rendering includes the rendering of the shading of the virtual object and the rendering of the shadow of the virtual object formed at a same time or at different times.

12. The method of claim 2 , wherein the rendering includes fusing the rendered shading and the rendered shadow to generate a rendering result.

13. The method of claim 1 , wherein the result of comparing comprises a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data of the rendered shading of the virtual object and the rendered shadow of the virtual object.

14. The method of claim 1 , wherein a spherical partial structure corresponding to the reference object and a flat partial structure corresponding to the reference plane are combined to form the combined structure.

15. A processor-implemented method for light estimation, comprising:

estimating light information corresponding to an input image using a light estimation model;

detecting a reference object in the input image;

determining object information of the reference object and plane information of a reference plane supporting the reference object;

rendering, based on the light information, the object information, and the plane information, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

training the light estimation model by updating the light estimation model based on a result of comparing the reference object and the rendered virtual object,

wherein the determining comprises, in response to the reference object being determined as an unknown object, determining the object information based on information of a predetermined proxy object.

16. An apparatus for light estimation, comprising:

one or more processors configured to:

estimate light information corresponding to an input image using a light estimation model;

detect a reference object in the input image;

determine object information of the reference object and plane information of a reference plane supporting the reference object;

render, based on the light information, the object information, and the plane information, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

train the light estimation model by updating the light estimation model based on a result of comparing the reference object and the rendered virtual object,

wherein the result of comparing comprises a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data of the rendered shading of the virtual object and the rendered shadow of the virtual object.

17. The apparatus of claim 16 , wherein, for the rendering, the one or more processors are further configured to:

determine shadow information of each sampling point of the reference plane by fusing light information and visibility information of each sampling point; and

perform the rendering of the shadow of the virtual object by projecting the shadow information of each sampling point of the reference plane onto a capture view of the input image.

18. The apparatus of claim 16 , wherein, for the determining, the one or more processors are further configured to determine the object information based on information of a predetermined proxy object.

19. The apparatus of claim 16 , wherein, for the updating, the one or more processors are further configured to update the light estimation model to reduce a difference between the reference object and the virtual object.

20. An electronic device, comprising:

a camera configured to generate an input image; and

one or more processors configured to:

estimate light information corresponding to the input image using a light estimation model;

detect a reference object in the input image;

determine object information of the reference object and plane information of a reference plane supporting the reference object;

render a virtual object corresponding to the reference object, a shading of the virtual object, and a shadow of the virtual object based on the light information, the object information, and the plane information; and

train the light estimation model by updating the light estimation model based on a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data representing the shading of the rendered virtual object and the shadow of the rendered virtual object.

21. The device of claim 20 , wherein, for the determining, the one or more processors are further configured to determine the object information based on information of a predetermined proxy object.

22. A processor-implemented method for light estimation, comprising:

obtaining a second image; and

estimating second light information corresponding to the second image using a trained light estimation model,

wherein the light estimation model is trained by:

estimating light information corresponding to an input image using a light estimation model;

detecting a reference object in the input image;

determining object information of the reference object and plane information of a reference plane supporting the reference object;

rendering, based on the light information, the object information, and the plane information, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

updating the light estimation model based on the rendered shading and shadow of the virtual object and based on a result of comparing the reference object and the virtual object,

wherein the result of comparing comprises a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data of the rendered shading of the virtual object and the rendered shadow of the virtual object.

23. The method of claim 22 , further comprising:

rendering a second virtual object corresponding to a second object in the second image, based on the second light information; and

superimposing the second virtual object on the second image to generate an augmented reality (AR).

24. A processor-implemented method for light estimation, comprising:

estimating light information corresponding to an input image using a light estimation model;

determining information of a reference object in the input image based on whether stored information corresponds to the reference object;

rendering, based on the light information and the information of the reference object, a virtual object corresponding to the reference object and a shading of the virtual object and/or a shadow of the virtual object; and

training the light estimation model by updating the light estimation model based on the rendered shading and shadow of the virtual object and based on a result of comparing the reference object and the rendered virtual object,

wherein the result of comparing comprises a result of comparing pixel data representing a shading of the reference object and a shadow of the reference object and pixel data of the rendered shading of the virtual object and the rendered shadow of the virtual object.

25. The method of claim 24 , wherein

the stored information is of a predetermined object, and

the determining comprises determining the stored information to be the information of the reference object in response to the reference object corresponding to the predetermined object.

26. The method of claim 24 , wherein the determining comprises, in response to the stored information corresponding to the reference object, determining the stored information to be the information of the reference object.

27. The method of claim 24 , wherein the information of the reference object comprises either one or both of object information of the reference object and plane information of a reference plane supporting the reference object.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (26)