Granted Patent B2

US 12,450,696 · App. 17/383,346 · Granted Oct 21, 2025

Inventors: Heewon Kim (Seoul, KR); Seon Min Rhee (Seoul, KR); Jihye Kim (Anyang-si, KR); Seungju Han (Seoul, KR)

Assignee: Samsung Electronics Co., Ltd.

G06T5/70G06N3/045G06N3/08G06T5/50G06T5/73G06T2207/10028G06T2207/20081G06T2207/20084

Abstract

Claims (80)

1. A processor-implemented image processing method, comprising:

generating a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and

generating, using a first neural network, a second depth image by applying the first depth image and the first surface normal image as input to the first neural network, where the first neural network generates the second depth image by changing the first depth image based on an embedding vector that comprises a feature of the first depth image and a feature of the first surface normal image such that the second depth image has an improved quality compared to the first depth images,

wherein the first neural network removes noise included in the first depth image and increases a sharpness of the first depth image.

2. The method of claim 1 , wherein the first depth image comprises a time-of-flight (TOF) image.

3. The method of claim 1 , further comprising:

generating a second surface normal image comprising surface normal vectors corresponding to pixels corresponding to the second depth image; and

applying the second depth image and the second surface normal image to a second neural network, and acquiring a third depth image by changing the second depth image using the second neural network,

wherein the second neural network generates the third depth image having a degraded quality compared to the second depth image, based on an embedding vector that comprises a feature of the second depth image and a feature of the second surface normal image.

4. The method of claim 3 , further comprising:

generating training data by pairing the second depth image and the third depth image.

5. The method of claim 3 , wherein the first neural network and the second neural network are trained based on a cycle-generative adversarial network (GAN) training method.

6. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

7. A training method, comprising:

acquiring a second depth image of a second type by applying a first depth image of a first type and a first surface normal image corresponding to the first depth image to a first neural network;

acquiring a third depth image of the first type by applying the acquired second depth image and a second surface normal image corresponding to the second depth image to a second neural network;

acquiring a first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network;

acquiring a first cycle consistency loss related to a difference between the third depth image and the first depth image; and

training the first neural network based on the first GAN loss and the first cycle consistency loss, wherein the trained first neural network generates an output depth image by changing an input depth image based on an embedding vector that comprises a feature of the input depth image and a feature of a corresponding surface normal image so that the output depth image has an improved quality compared to the input depth image,

wherein the corresponding surface normal image is generated corresponding to pixels of the input depth image, and

wherein the first neural network removes noise included in the input depth image and increases a sharpness of the input depth image.

8. The training method of claim 7 , wherein the training of the first neural network comprises:

acquiring a fifth depth image of the first type by applying a fourth depth image of the second type and a fourth surface normal image corresponding to the fourth depth image to the second neural network;

acquiring a sixth depth image of the second type by applying the acquired fifth depth image and a fifth surface normal image corresponding to the fifth depth image to the first neural network;

acquiring a second GAN loss related to whether the fifth depth image is an image generated by the second neural network;

acquiring a second cycle consistency loss related to a difference between the sixth depth image and the fourth depth image; and

training the first neural network and the second neural network based on the first GAN loss, the second GAN loss, the first cycle consistency loss, and the second cycle consistency loss.

9. The training method of claim 8 , further comprising:

acquiring a first identity (ID) loss related to a difference between the first depth image and a depth image acquired by applying the first depth image and the first surface normal image to the second neural network;

acquiring a second ID loss related to a difference between the fourth depth image and a depth image acquired by applying the fourth depth image and the fourth surface normal image to the first neural network; and

training the first neural network and the second neural network further based on the first ID loss and the second ID loss.

10. The training method of claim 8 , further comprising:

training the first neural network and the second neural network further based on a first range preserving loss for limiting a difference between the first depth image and the second depth image, and a second range preserving loss for limiting a difference between the fourth depth image and the fifth depth image.

11. The training method of claim 8 , further comprising:

training the second neural network based on a loss for discriminating a difference between the fourth depth image and the fifth depth image and a difference between the fourth depth image and a real depth image of the first type corresponding to the fourth depth image.

12. The training method of claim 7 , wherein

the first neural network comprises a pre-trained neural network trained to generate a depth image of the second type by changing a depth image of the first type in a first direction, and

the second neural network comprises a pre-trained neural network trained to generate a depth image of the first type by changing a depth image of the second type in a second direction opposite to the first direction.

13. The training method of claim 7 , wherein a depth image of the first type comprises a depth image with noise,

wherein a depth image of the second type comprises a depth image that does not contain noise,

wherein the first neural network is configured to generate a depth image having noise removed by removing noise from an input depth image, and

wherein the second neural network is configured to generate a depth image having noise added by adding noise to another input depth image.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 7 .

15. An apparatus with image processing, comprising:

at least one processor;

at least one memory storing instructions, which when executed by the at least one processor configure the at least one processor to:

generate a first surface normal image comprising surface normal vectors corresponding to pixels of a first depth image; and

generate, using a first neural network, a second depth image by applying the first depth image and the first surface normal image as input to the first neural network, where the first neural network generates the second depth image by changing the first depth image based on an embedding vector that comprises a feature of the first depth image and a feature of the first surface normal image such that the second depth image has an improved quality compared to the first depth image,

wherein the first neural network is further configured to remove noise included in the first depth image and increase a sharpness of the first depth image.

16. The apparatus of claim 15 , wherein the first depth image comprises a time-of-flight (TOF) image.

17. The apparatus of claim 15 , wherein the processor is further configured to:

generate a second surface normal image comprising surface normal vectors corresponding to pixels corresponding to the second depth image;

apply the second depth image and the second surface normal image to a second neural network; and

acquire a third depth image by changing the second depth image using the second neural network, and

wherein the second neural network is configured to generate the third depth image having a degraded quality compared to the second depth image, based on an embedding vector that comprises a feature of the second depth image and a feature of the second surface normal image.

18. The apparatus of claim 17 , wherein the processor is further configured to generate training data by pairing the second depth image and the third depth image.

19. The apparatus of claim 17 , wherein the first neural network and the second neural network are trained based on a cycle-generative adversarial network (GAN) training method.

20. An apparatus with image processing, comprising:

at least one processor:

at least one memory storing instructions, which when executed by the at least one processor configure the at least one processor to:

acquire a second depth image having weak noise by applying a first depth image having strong noise and a first surface normal image corresponding to the first depth image to a first neural network;

acquire a third depth image having strong noise by applying the acquired second depth image and a second surface normal image corresponding to the second depth image to a second neural network;

acquire a first generative adversarial network (GAN) loss related to whether the second depth image is an image generated by the first neural network;

acquire a first cycle consistency loss related to a difference between the third depth image and the first depth image; and

train the first neural network based on the first GAN loss and the first cycle consistency loss;

wherein the trained first neural network generates an output depth image by changing an input depth image based on an embedding vector that comprises a feature of the input depth image and a feature of corresponding surface normal image so that the output depth image has an improved quality compared to the input depth image,

wherein the corresponding surface normal image is generated corresponding to pixels of the input depth image, and

wherein the first neural network removes noise included in the input depth image and increases a sharpness of the input depth image.

21. The apparatus of claim 20 , wherein the training of the first neural network comprises:

acquiring a fifth depth image having strong noise by applying a fourth depth image having weak noise and a fourth surface normal image corresponding to the fourth depth image to the second neural network;

acquiring a sixth depth image having weak noise by applying the acquired fifth depth image and a fifth surface normal image corresponding to the fifth depth image to the first neural network;

acquiring a second GAN loss related to whether the fifth depth image is an image generated by the second neural network;

acquiring a second cycle consistency loss related to a difference between the sixth depth image and the fourth depth image; and

training the first neural network and the second neural network based on the first GAN loss, the second GAN loss, the first cycle consistency loss, and the second cycle consistency loss.

22. The apparatus of claim 21 , wherein the at least one processor is further configured to:

acquire a first identity (ID) loss related to a difference between the first depth image and a depth image acquired by applying the first depth image and the first surface normal image to the second neural network;

acquire a second ID loss related to a difference between the fourth depth image and a depth image acquired by applying the fourth depth image and the fourth surface normal image to the first neural network; and

train the first neural network and the second neural network further based on the first ID loss and the second ID loss.

23. The apparatus of claim 21 , wherein the at least one processor is further configured to train the first neural network and the second neural network further based on a first range preserving loss for limiting a difference between the first depth image and the second depth image, and a second range preserving loss for limiting a difference between the fourth depth image and the fifth depth image.

24. The apparatus of claim 21 , wherein the at least one processor is further configured to train the second neural network based on a loss for discriminating a difference between the fourth depth image and the fifth depth image and a difference between the fourth depth image and a real depth image of the first type corresponding to the fourth depth image.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (29)