Augmented reality (AR) device and method of predicting pose therein

1. A method of predicting a pose in an augmented reality (AR) device, the method comprising:

obtaining, using an inertial measurement unit (IMU) sensor provided in the AR device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency;

estimating intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency by a camera provided in the AR device;

generating, by a processor, a pose prediction model for predicting relative 6D poses of the AR device;

inputting the IMU values as a first input to the pose prediction model, and separately inputting the intermediate 6D poses, which are estimated prior to predicting the relative 6D, as a second input to the pose prediction model; and

outputting the relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network implemented by the pose prediction model.

2. The method of claim 1 , wherein the intermediate 6D poses comprise:

first type pose data of a frame rate estimated by using a simultaneous localization and mapping (SLAM) technique of fusing the obtained IMU values and the images obtained at the frame rate of the second frequency, or

second type pose data of the IMU rate of the first frequency including pose data corresponding to integral values calculated by a pre-integration method from the obtained IMU values and image based pose data of the frame rate estimated by using the SLAM technique.

3. The method of claim 1 , wherein the pose prediction model outputs relative 6D poses, each corresponding to a window moving in a sliding manner, according to the learning of the deep neural network based on k IMU values, from among the IMU values, and m intermediate 6D poses, from among the intermediate 6D poses, and

wherein k is a natural number and m is a natural number in the window.

4. The method of claim 3 , wherein the window slides based on the IMU rate and the k IMU values and the m intermediate 6D poses corresponding to the sliding window is input to the deep neural network, and

the pose prediction model outputs the relative 6D pose corresponding to each window at the IMU rate.

5. The method of claim 1 , further comprising updating and setting the estimated intermediate 6D pose to a ground truth (GT) pose at a real time point when an image is photographed, based on estimation on the intermediate 6D pose being completed.

6. The method of claim 5 , wherein the pose prediction model performs the learning using the deep neural network via a self-supervised method based on updating of the GT pose.

7. The method of claim 1 , wherein the pose prediction model performs the learning using the deep neural network with multi-rate inputs including the IMU values at the IMU rate of the first frequency and the intermediate 6D pose at the frame rate of the second frequency, and

the first frequency is greater than the second frequency.

8. The method of claim 1 , wherein the deep neural network comprises a recurrent neural network using a long short term memory or a gated recurrent unit.

9. The method of claim 8 , wherein the deep neural network performs back propagation through time by reducing a loss function using an updated ground truth (GT) pose.

10. The method of claim 1 , further comprising, based on newly obtained IMU values and newly estimated intermediate 6D poses according to the movement of the AR device being input to the learned deep neural network of the pose prediction model, outputting a result of an inference of predicting relative 6D poses in real time through processing by the learned deep neural network of the pose prediction model.

11. The method of claim 1 , wherein the same IMU values used to estimate the intermediate 6D poses are input as the first input to the pose prediction model, and

wherein the first input and the second input are input in parallel to the pose prediction model.

12. A non-transitory computer readable recording medium having recorded thereon a program for executing, in a computer, a method of predicting a pose in an augmented reality (AR) device, the method comprising:

obtaining, using an inertial measurement unit (IMU) sensor provided in the AR device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency;

estimating intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency by a camera provided in the AR device;

generating, by a processor, a pose prediction model for predicting relative 6D poses of the AR device;

inputting the IMU values as a first input to the pose prediction model, and separately inputting the intermediate 6D poses, which are estimated prior to predicting the relative 6D, as a second input to the pose prediction model; and

outputting the relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network implemented by the pose prediction model.

13. An augmented reality (AR) device comprising:

an inertial measurement unit (IMU) sensor configured to obtain IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency;

a camera configured to obtain images surrounding the AR device at a frame rate of a second frequency;

a visual-inertial simultaneous localization and mapping (VI-SLAM) module configured to estimate intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and the images surrounding the AR device; and

a processor configured to generate a pose prediction model for predicting relative 6D poses of the AR device,

wherein the IMU values are input as a first input to the pose prediction model, and the intermediate 6D poses, which are estimated prior to predicting the relative 6D, are separately input as a second input to the pose prediction model, and

wherein the relative 6D poses of the AR device are output by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network implemented by the pose prediction model.

14. The AR device of claim 13 , wherein the pose prediction model outputs the relative 6D poses, each corresponding to a window moving in a sliding manner, according to the learning of the deep neural network based on k IMU values, from among the IMU values, and m intermediate 6D poses, from among the intermediate 6D poses, and

wherein k is a natural number and m is a natural number in the window.

15. The AR device of claim 14 , wherein the window slides based on the IMU rate and the k IMU values and the m intermediate 6D poses corresponding to the sliding window is input to the deep neural network, and

the pose prediction model outputs the relative 6D pose corresponding to each window at the IMU rate.

16. The AR device of claim 13 , wherein the processor is further configured to update and set the estimated intermediate 6D pose to a ground truth (GT) pose at a real time point when an image is photographed, based on estimation on the intermediate 6D pose by the VI-SLAM module being completed.

17. The AR device of claim 16 , wherein the pose prediction model performs the learning using the deep neural network via a self-supervised method based on updating of the GT pose.

18. The AR device of claim 13 , wherein the pose prediction model performs the learning using the deep neural network with multi-rate inputs including the IMU values at the IMU rate of the first frequency and the intermediate 6D pose at the frame rate of the second frequency, and

the first frequency is greater than the second frequency.

19. The AR device of claim 13 , wherein the deep neural network comprises a recurrent neural network using a long short term memory or a gated recurrent unit.

20. The AR device of claim 19 , wherein the deep neural network performs back propagation through time by reducing a loss function using an updated GT pose.

21. The AR device of claim 13 , wherein the processor is further configured to output a result of inference of predicting relative 6D poses in real time through processing by the learned deep neural network of the pose prediction model, when newly obtained IMU values and newly estimated intermediate 6D poses according to the movement of the AR device are input to the learned deep neural network of the pose prediction model.

22. A method of predicting a pose in a processor, the method comprising:

obtaining, by using an inertial measurement unit (IMU) sensor provided in an augmented reality (AR) device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency;

obtaining intermediate 6-degrees of freedom (6D) poses of the AR device estimated based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency using a camera provided in the AR device;

generating a pose prediction model for predicting relative 6D poses of the AR device; and

inputting the IMU values as a first input to the pose prediction model, and separately inputting the intermediate 6D poses, which are obtained prior to predicting the relative 6D, as a second input to the pose prediction model; and

outputting the relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network implemented by the pose prediction model.