Augmented reality device for obtaining depth information and method of operating the same
Provided are an augmented reality device and a method of operating the same. The augmented reality device adjusts a depth value based on a gravity direction measured by an IMU sensor in order to obtain a depth map having a high accuracy without an additional hardware module. The augmented reality device obtains a depth map from an image obtained using the camera, obtains a normal vector of at least one pixel included in the depth map, modifies a direction of the normal vector of the at least one pixel, based on the gravity direction measured by the IMU sensor, and adjusts a depth value of the at least one pixel based on the modified direction of the normal vector.
1 . An augmented reality device comprising:
at least one camera;
an inertial measurement unit (IMU) sensor configured to measure a gravity direction;
a memory storing at least one instruction; and
at least one processor configured to execute the at least one instruction to:
obtain a left-eye image and a right-eye image by using the at least one camera,
obtain a depth map by calculating a disparity between the left-eye image and the right-eye image by applying the left-eye image and the right-eye image as inputs to an artificial intelligence (AI) model,
obtain a normal vector of at least one pixel included in the depth map,
modify a direction of the normal vector of the at least one pixel, based on the gravity direction measured by the IMU sensor,
obtain a loss of the depth map obtained by the AI model, based on depth values of a first pixel on a plane defined by the normal vector with the modified direction and a plurality of pixels that are adjacent to the first pixel, and
adjust a depth value of the at least one pixel, through the AI model trained by applying the obtained loss.
2 . The augmented reality device of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
convert the at least one pixel into a three-dimensional (3D) coordinate value, based on a direction vector of the at least one pixel included in the depth map and the depth value of the at least one pixel, and
obtain the normal vector by obtaining a cross-product of the 3D coordinate value of the plurality of pixels that are adjacent to the at least one pixel in at least one of an upper direction, a lower direction, a left direction, and a right direction.
3 . The augmented reality device of claim 2 , wherein the at least one camera comprises a left-eye camera configured to obtain the left-eye image and a right-eye camera configured to obtain the right-eye image, and
wherein the at least one processor is further configured to execute the at least one instruction to:
apply the left-eye image and the right-eye image as inputs to the AI model, and
obtain the depth map by obtaining the disparity according to a similarity in intensity values between pixels in the left-eye image and pixels in the right-eye image by using the AI model.
4 . The augmented reality device of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
define the plane to include a second pixel having the normal vector with the modified direction and the plurality of pixels that are adjacent to the second pixel,
obtain the depth values of the plurality of pixels based on a plurality of points where the defined plane and a ray vector of the at least one camera meet each other,
obtain a difference value between the depth value of the second pixel and each of the depth values of the plurality of pixels, and
obtain the loss by performing a weighted sum in which a weight is applied to the difference value obtained with respect to each of the plurality of pixels.
5 . The augmented reality device of claim 4 , wherein the weight comprises a first weight determined based on a distance between each of the plurality of pixels in the depth map and the at least one camera, and a second weight determined based on a difference in intensity values between the first pixel of the depth map and each of the plurality of pixels.
6 . The augmented reality device of claim 5 , wherein the at least one processor is further configured to execute the at least one instruction to obtain an adjusted depth map by performing inference in which the left-eye image and the right-eye image are provided to the trained AI model.
7 . The augmented reality device of claim 1 , wherein the at least one camera comprises a left-eye camera configured to obtain the left-eye image and a right-eye camera configured to obtain the right-eye image, and
wherein the at least one processor is further configured to execute the at least one instruction to:
modify the direction of the normal vector in the left-eye image and the right-eye image according to the gravity direction or a direction perpendicular to the gravity direction,
perform a plane hypothesis along the modified direction of the normal vector or a direction perpendicular to the modified direction of the normal vector,
obtain the depth value of the at least one pixel by performing a plane sweep along a plane defined through the plane hypothesis, and
modify the depth value of the at least one pixel by using the depth value.
8 . The augmented reality device of claim 1 , wherein the at least one camera comprises a time-of-flight (ToF) camera, and
wherein the at least one processor is further configured to execute the at least one instruction to obtain the depth map by using the ToF camera.
9 . The augmented reality device of claim 8 , wherein the at least one processor is further configured to execute the at least one instruction to:
define a plane for each pixel of the at least one pixel, based on the normal vector with the modified direction,
identify a plane region of the plane defined in the depth map, based on regions segmented according to color information of an RGB image, and
adjust the depth value, based on depth values of pixels in the plane region.
10 . A method, performed by an augmented reality device, of adjusting a depth value, the method comprising:
obtaining a left-eye image and a right-eye image by using at least one camera;
obtaining a depth map by calculating a disparity between the left-eye image and the right-eye image by applying the left-eye image and the right-eye image as inputs to an artificial intelligence (AI) model;
obtaining a normal vector of at least one pixel included in the depth map;
modifying a direction of the normal vector of the at least one pixel based on a gravity direction measured by an inertial measurement unit (IMU) sensor;
obtaining a loss of the depth map obtained by the AI model, based on depth values of a first pixel on a plane defined by the normal vector with the modified direction and a plurality of pixels that are adjacent to the first pixel; and
adjusting the depth value of the at least one pixel, through the AI model trained by applying the obtained loss.
11 . The method of claim 10 , wherein the obtaining the depth map comprises:
providing the left-eye image obtained by using a left-eye camera and the right-eye image obtained by using a right-eye camera to the AI model; and
obtaining the depth map by obtaining a disparity according to a similarity in intensity values between pixels in the left-eye image and pixels in the right-eye image by using the AI model.
12 . The method of claim 10 , wherein the calculating of the loss comprises:
defining the plane to include a second pixel having the normal vector with the modified direction and the plurality of pixels that are adjacent to the second pixel;
obtaining depth values of the plurality of pixels based on a plurality of points where the defined plane and a ray vector of the at least one camera meet each other;
obtaining a difference value between the depth value of the second pixel and each of the depth values of the plurality of pixels; and
obtaining the loss by performing a weighted sum in which a weight is applied to the difference value obtained with respect to each of the plurality of pixels.
13 . The method of claim 12 , wherein the weight comprises a first weight determined based on a distance between each of the plurality of pixels in the depth map and the at least one camera, and a second weight determined based on a difference in intensity values between the first pixel of the depth map and each of the plurality of pixels.
14 . The method of claim 13 , further comprising obtaining an adjusted depth map by performing inference in which the left-eye image and the right-eye image are provided to the trained AI model.
15 . The method of claim 10 , wherein the modifying the direction of the normal vector for each pixel comprises:
modifying the direction of the normal vector of the left-eye image obtained by using a left-eye camera and the right-eye image obtained by using a right-eye camera according to the gravity direction or a direction perpendicular to the gravity direction, and
wherein the adjusting of the depth value of the at least one pixel comprises:
performing a plane hypothesis along the modified direction of the normal vector or a direction perpendicular to the modified direction of the normal vector;
obtaining the depth value by performing a plane sweep along the plane defined through the plane hypothesis; and
adjusting the depth value of the at least one pixel by using the depth value.
16 . The method of claim 10 , wherein the obtaining the depth map comprises obtaining the depth map by using a time-of-flight camera.
17 . The method of claim 16 , wherein the modifying the depth value of the at least one pixel comprises:
defining a plane for each pixel of the at least one pixel, based on the normal vector with the modified direction;
identifying a plane region of the plane defined in the depth map based on regions segmented according to color information of an RGB image; and
adjusting the depth value of the at least one pixel, based on depth values of adjacent pixels in the plane region.
18 . A non-transitory computer-readable storage medium comprising instructions that are executed by an augmented reality device to perform a method comprising:
obtaining a left-eye image and a right-eye image by using at least one camera;
obtaining a depth map by calculating a disparity between the left-eye image and the right-eye image by applying the left-eye image and the right-eye image as inputs to an artificial intelligence (AI) model;
obtaining a normal vector of at least one pixel included in the depth map;
modifying a direction of the normal vector of the at least one pixel based on a gravity direction measured by an inertial measurement unit sensor;
obtaining a loss of the depth map obtained by the AI model, based on depth values of a first pixel on a plane defined by the normal vector with the modified direction and a plurality of pixels that are adjacent to the first pixel; and
adjusting a depth value of the at least one pixel, through the AI model trained by applying the obtained loss.