Electronic device and operating method of electronic device
View Patent ↗Disclosed is an operating method of an electronic device which includes a processor performing machine learning of a monocular depth estimation module. The operating method includes obtaining, by the processor, a first image and a second image respectively photographed by a first camera and a second camera of different locations, inferring, by the processor, a plurality of multi-cyclic disparities by applying weights of the monocular depth estimation module to the first image plural times and calculating a plurality of multi-cyclic loss functions based on the first image, the second image, and the plurality of multi-cyclic disparities, and updating, by the processor, the weights of the monocular depth estimation module through machine learning, based on the plurality of multi-cyclic loss functions.
1. An operating method of an electronic device which includes a processor configured to train a monocular depth estimation module, the method comprising:
obtaining, by the processor, a first image photographed by a first camera and a second image photographed by a second camera;
inferring, by the processor, a plurality of multi-cyclic disparities by applying weights of the monocular depth estimation module to the first image a plurality of times and calculating a plurality of multi-cyclic loss functions based on the first image, the second image, and the plurality of multi-cyclic disparities; and
updating, by the processor, the weights of the monocular depth estimation module through the training based on the plurality of multi-cyclic loss functions,
wherein the calculating of the plurality of multi-cyclic loss functions includes
inferring a (k+1)-th multi-cyclic disparity of the plurality of multi-cyclic disparities from a k-th restored image by applying the weights of the monocular depth estimation module,
generating a (k+1)-th restored image from the second image by using the (k+1)-th multi-cyclic disparity, and
calculating a difference between the first image and the (k+1)-th restored image as at least another of the multi-cyclic loss functions, and
wherein k is a positive integer.
2. The method of claim 1 , wherein the plurality of multi-cyclic disparities are respectively inferred with respect to pixels of the first image.
3. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions further includes, prior to the inferring the (k+1)-th multi-cycle disparity:
inferring a first multi-cyclic disparity of the plurality of multi-cyclic disparities from the first image by applying the weights of the monocular depth estimation module;
generating a first restored image from the second image using the first multi-cyclic disparity; and
calculating a difference between the first image and the first restored image as a first multi-cyclic loss function of the plurality of multi-cyclic loss functions.
4. The method of claim 1 , wherein the updating of the weights of the monocular depth estimation module through the training includes:
updating the weights of the monocular depth estimation module through the training such that differences between the plurality of multi-cyclic disparities decrease.
5. The method of claim 1 , wherein the updating of the weights of the monocular depth estimation module through the training includes:
updating the weights of the monocular depth estimation module through the training such that a sum of the plurality of multi-cyclic disparities decreases.
6. The method of claim 1 , further comprising:
obtaining, by the processor, a third image photographed by a third camera and a fourth image photographed by a fourth camera;
inferring, by the processor, a plurality of second multi-cyclic disparities by applying weights of the monocular depth estimation module to the third image and calculating a plurality of second multi-cyclic loss functions based on the third image, the fourth image, and the plurality of second multi-cyclic disparities; and
updating, by the processor, the weights of the monocular depth estimation module through the training, based on the plurality of second multi-cyclic loss functions.
7. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions includes repeating the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image until the “k” reaches a maximum value while sequentially increasing the “k”.
8. The method of claim 7 , wherein the maximum value of the “k” is set by a user.
9. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions includes repeating the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image while sequentially increasing the “k” until at least one of the plurality of multi-cyclic loss functions diverges.
10. The method of claim 9 , wherein the weights of the monocular depth estimation module are initialized in response to the at least one of the plurality of multi-cyclic loss functions diverging.
11. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions further includes:
repeating, at least once, the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image; and
calculating differences between the (k+1)-th restored images as others of the multi-cyclic loss functions.
12. An electronic device comprising:
a first camera;
an interface device configured to communicate with an external database; and
a processor configured to execute a monocular depth estimation module, the processor configured to
obtain a first image and a second image from the external database through the interface device,
infer a plurality of multi-cyclic disparities corresponding to pixels of the second image by applying weights of the monocular depth estimation module to the first image and calculate a plurality of multi-cyclic loss functions corresponding to the pixels of the second image based on the first image, the second image, and the plurality of multi-cyclic disparities, and
update the weights of the monocular depth estimation module through machine learning based on the plurality of multi-cyclic loss functions, and
wherein the processor is further configured to
obtain a third image from the first camera,
infer disparities by applying the weights of the monocular depth estimation module to the third image, and
calculate depth data of the third image from the disparities,
wherein, while sequentially increasing “k” (k being a positive integer), the processor is configured to repeat operations
calculating a (k+1)-th multi-cyclic disparity from a k-th restored image by using the weights of the monocular depth estimation module,
generating a (k+1)-th restored image from the first image by using the (k+1)-th multi-cyclic disparity,
calculating a (k+1)-th multi-cyclic loss function based on a difference between the second image and the (k+1)-th restored image and differences between the first to k-th multi-cyclic disparities, and
updating the weights of the monocular depth estimation module through the machine learning based on the (k+1)-th multi-cyclic loss function.
13. The electronic device of claim 12 , further comprising:
a second camera,
wherein the processor is further configured to
obtain a fourth image from the second camera, respectively,
infer a plurality of second multi-cyclic disparities respectively corresponding to pixels of the third image by applying weights of the monocular depth estimation module to the fourth image and calculate a plurality of second multi-cyclic loss functions respectively corresponding to the pixels of the third image based on the third image, the fourth image, and the plurality of second multi-cyclic disparities, and
update the weights of the monocular depth estimation module through the machine learning based on the plurality of second multi-cyclic loss functions.
14. The electronic device of claim 12 , wherein the processor is further configured to, prior to the inferring the (k+1)-th multi-cycle disparity,
calculate a first multi-cyclic disparity from the first image using the monocular depth estimation module,
generating a first restored image from the first image using the first multi-cyclic disparity,
calculate a first multi-cyclic loss function based on a difference between the second image and the first restored image, and
perform machine learning of the monocular depth estimation module based on the first multi-cyclic loss function.
15. The electronic device of claim 14 , wherein the processor is further configured to
calculate differences between the first to (k+1)-th restored images as an additional (k+1)-th multi-cyclic loss function, and
update the weights of the monocular depth estimation module through the machine learning based on the additional (k+1)-th multi-cyclic loss function.
16. The electronic device of claim 12 , wherein the processor is further configured to
calculate differences between the first to (k+1)-th restored images as an additional (k+1)-th multi-cyclic loss function, and
update the weights of the monocular depth estimation module through the machine learning based on the additional (k+1)-th multi-cyclic loss function.
17. The electronic device of claim 12 , wherein the processor is further configured to initialize the monocular depth estimation module in response to the (k+1)-th multi-cyclic loss function diverging.
18. An electronic device comprising:
a camera; and
a processor configured to execute a monocular depth estimation module,
wherein the processor is configured to
obtain an image from the camera,
infer disparities from the image using the monocular depth estimation module, and
generate depth data of the image from the disparities, and
wherein the monocular depth estimation module is trained based on a plurality of multi-cyclic disparities and a plurality of multi-cyclic loss functions respectively generated from pixels, and
wherein the plurality of multi-cyclic loss functions are generated by
inferring a (k+1)-th multi-cyclic disparity of the plurality of multi-cyclic disparities from a k-th restored image by applying weights of the monocular depth estimation module,
generating a (k+1)-th restored image from a second image by using the (k+1)-th multi-cyclic disparity, and
calculating a difference between a first image and the (k+1)-th restored image as at least another of the multi-cyclic loss functions, and
wherein k is a positive integer.