Device and method with pose estimator based on current predicted motion state array

1. A processor-implemented method of a device estimating a pose of the device, the method comprising:

generating, using an inertial measurement unit (IMU) of the device, IMU data;

determining a first pose of the device a first time point based on the IMU data;

generating a current predicted motion state array based on the IMU data; and

estimating an M-th predicted pose of the device at an M-th time point after the first time point based on the current predicted motion state array,

wherein M denotes a natural number greater than 1,

wherein the generating of the current predicted motion state array comprises:

smoothing the IMU data;

generating a first motion state vector at the first time point based on the smoothed IMU data;

determining an N-th predicted motion state vector at an N-th time point based on the first motion state vector, wherein N denotes a natural number greater than M; and

generating the current predicted motion state array based on a motion model, a previous predicted motion state array, and the N-th predicted motion state vector.

2. The method of claim 1 , wherein the generating of the first motion state vector comprises:

generating the first motion state vector by correcting the smoothed IMU data based on a difference between the first pose and a first predicted pose that is estimated in advance.

3. The method of claim 1 , wherein the determining of the N-th predicted motion state vector comprises:

generating the N-th predicted motion state vector using a long-short term memory (LSTM) recurrent neural network (RNN) using the first motion state vector as an input.

4. The method of claim 1 , wherein the generating of the first motion state vector comprises:

determining a frequency of generation of the first motion state vector based on at least one of an ocular movement velocity of a user of the device or a head movement velocity of the user.

5. The method of claim 1 , further comprising:

determining whether the device is in a static state based on the IMU data,

wherein the determining of the first pose of the device comprises:

determining the first pose based on whether the device is in the static state.

6. The method of claim 1 , further comprising:

generating a virtual object to be output at the M-th time point; and

outputting the virtual object.

7. The method of claim 6 , further comprising:

adjusting the virtual object based on a pose change of the device occurring during a period for rendering of the virtual object.

8. The method of claim 6 , wherein the outputting of the virtual object comprises:

outputting the virtual object such that an image at the M-th time point and the virtual object are synthesized.

9. The method of claim 6 , wherein the virtual object is a holographic image.

10. The method of claim 1 , further comprising:

generating a map of a space around the device;

determining an initial pose of the device using the generated map; and

determining whether the device is in a static state based on the IMU data,

wherein the first pose is determined when the device is in the static state.

11. The method of claim 10 , wherein the generating of the map comprises:

generating a left image using a left camera of the device;

generating one or more first feature points from the left image;

generating a right image using a right camera of the device;

generating one or more second feature points from the right image;

matching the first feature points and the second feature points; and

generating the map based on matched feature points and unmatched feature points between the first feature points and the second feature points.

12. The method of claim 10 , wherein the determining of the initial pose comprises:

generating one or more feature points from an image generated by at least one of a left camera or a right camera of the device;

determining a target feature point corresponding to the one or more feature points among preset candidate feature points in the map; and

determining the initial pose based on the target feature point.

13. The method of claim 12 , wherein the determining of the initial pose further comprises:

determining a remaining feature point that does not correspond to the preset candidate feature points in the map among the one or more feature points; and

updating the map by adding, to the map, a feature point corresponding to the remaining feature point.

14. The method of claim 1 , wherein the device is a wearable device.

15. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

16. A device for estimating a pose of the device, comprising:

at least one camera;

an inertial measurement unit (IMU); and

a processor configured to:

generate IMU data using the IMU of the device;

determine a first pose of the device based on the IMU data;

generate a current predicted motion state array based on the IMU data; and

estimate an M-th predicted pose at an M-th time point after a first time point of the first pose based on the current predicted motion state array, wherein M denotes a natural number greater than 1,

wherein the processor is configured to generate the current predicted motion state array by:

smoothing the IMU data;

generating a first motion state vector at the first time point based on the smoothed IMU data;

determining an N-th predicted motion state vector at an N-th time point based on the first motion state vector, wherein N denotes a natural number greater than M; and

generating the current predicted motion state array based on a motion model, a previous predicted motion state array, and the N-th predicted motion state vector.

17. The device of claim 16 , further comprising a memory storing instructions, which when executed by the processor, configure the processor to perform the generation of the IMU data, the determination of the first pose, the generation of the current predicted motion state array and the estimation of the M-th predicted pose.

18. A processor-implemented method, the method comprising:

determining a first pose of a device based on an inertial measurement unit (IMU) data at a first time point;

generating a current predicted motion state array based on the IMU data;

estimating a second pose of the device at a second time point after the first time point based on the current predicted motion state array;

generating a virtual object to be output at the second time point;

adjusting the virtual object based on a pose change of the device occurring during a period for rendering of the virtual object; and

outputting the virtual object.

19. The method of claim 18 , wherein the generating of the current predicted motion state array comprises:

generating a first motion state vector at the first time point based on the IMU data;

determining a predicted motion state vector at a third time point based on the first motion state vector; and

generating the current predicted motion state array based on a motion model, a previous predicted motion state array, and the predicted motion state vector.