Object detection and tracking

1. A non-transitory computer readable medium that embodies instructions that cause one or more processors to perform a method comprising:

(A) acquiring, from a camera, an image frame;

(B) acquiring, from an inertial sensor, a sensor data sequence;

(C) tracking a first pose of an object in a real scene based at least on the image frame;

(D) deriving a sensor pose of the object, using the inertial sensor, based on the sensor data sequence, where the sensor pose is a separate pose from the first pose;

(E) determining whether the first pose is lost such that the first pose is no longer located;

(F) retrieving from one or more memories, or generating from a 3D model stored in one or more memories, a training template corresponding to a view that is based on the sensor pose obtained on or after the first pose is lost such that the view is a view of the object just before the object pose is no longer tracked; and

(G) deriving a second pose of the object using the training template.

2. The non-transitory computer readable medium according to claim 1 ,

wherein (D) includes

predicting the sensor pose of the inertial sensor based on the sensor data sequence using a non-linear filter algorithm.

3. The non-transitory computer readable medium according to claim 2 ,

wherein the method further comprises:

obtaining feature positions of the object for the image frame,

wherein (F) includes

retrieving, from one or more memories, the training template containing information of (i) a view, (ii) template features obtained from an image of a 3D model rendered from the view and/or an image of a reference object captured from the view, and (iii) 3D points corresponding to the template features, where the 3D model and the reference object correspond to the object, and the view is based on the sensor pose obtained on or after the first pose is lost, and

wherein (G) includes

deriving the second pose of the object using the feature positions and the training template.

4. The non-transitory computer readable medium according to claim 3 ,

wherein obtaining the 3D points in the image frame includes obtaining the feature positions of the object based on the sensor pose.

5. The non-transitory computer readable medium according to claim 3 ,

wherein the method further comprises:

retrieving, from one or more memories, a spatial relationship between the camera and the inertial sensor; and

wherein the view is based on (i) the sensor pose obtained on or after the first pose is lost and (ii) the spatial relationship.

6. The non-transitory computer readable medium according to claim 1 , wherein

one or more training templates are stored in the one or more memories; and

each of the one or more training templates includes feature keypoints trained prior to a user having lost the first pose.

7. The non-transitory computer readable medium according to claim 1 , wherein

a template generated from the 3D model is trained with feature keypoints in real-time after continuous tracking of the first pose stops.

8. The non-transitory computer readable medium according to claim 1 , further comprising:

tracking the object based at least on the second pose after the first pose is lost.

9. The non-transitory computer readable medium according to claim 1 , wherein:

continuous tracking is determined to not be maintained based on an error function ƒ being greater than a first predefined threshold T (ƒ>T (or ƒ<T)).

10. The non-transitory computer readable medium according to claim 1 , wherein:

the view that is based on the sensor pose obtained on or after the first pose is lost is a first view;

a second view is a view at where tracking of the object started, which is located at a different location from the first view; and

the second pose of the object is at the second view.

11. A method comprising:

(A) acquiring, from a camera, an image frame;

(B) acquiring, from an inertial sensor, a sensor data sequence;

(C) tracking a first pose of an object in a real scene based at least on the image frame;

(D) deriving a sensor pose of the object, using the inertial sensor based on the sensor data sequence, where the sensor pose is a separate pose from the first pose;

(E) determining whether the first pose is lost such that the first pose is no longer located;

(F) retrieving from one or more memories, or generating from a 3D model stored in one or more memories, a training template corresponding to a view that is based on the sensor pose obtained on or after the first pose is lost such that the view is a view of the object just before the object pose is no longer tracked; and

(G) deriving a second pose of the object using the training template.

12. The method according to claim 11 ,

wherein (D) includes

predicting the sensor pose of the inertial sensor based on the sensor data sequence using a non-linear filter algorithm.

13. The method according to claim 12 ,

wherein the method further comprises:

obtaining feature positions of the object for the image frame,

wherein (F) includes

retrieving, from one or more memories, the training template containing information of (i) a view, (ii) template features obtained from an image of a 3D model rendered from the view and/or an image of a reference object captured from the view, and (iii) 3D points corresponding to the template features, where the 3D model and the reference object correspond to the object, and the view is based on the sensor pose obtained on or after the first pose is lost, and

wherein (G) includes

deriving the second pose of the object using the feature positions and the training template.

14. The method according to claim 13 ,

wherein obtaining the 3D points in the image frame includes obtaining the feature positions of the object based on the sensor pose.

15. The method according to claim 13 ,

wherein the method further comprises:

retrieving, from one or more memories, a spatial relationship between the camera and the inertial sensor; and

wherein the view is based on (i) the sensor pose obtained on or after the first pose is lost and (ii) the spatial relationship.

16. A head-mounted display device comprising:

a camera;

an inertial sensor fixed or adjustably fixed with respect to the camera with a spatial relationship; and

a processor configured for:

(A) acquiring, from the camera, an image frame;

(B) acquiring, from the inertial sensor, a sensor data sequence;

(C) tracking a first pose of an object in a real scene based at least on the image frame;

(D) deriving a sensor pose of the object, using the inertial sensor based on the sensor data sequence, where the sensor pose is a separate pose from the first pose;

(E) determining whether the first pose is lost such that the first pose is no longer located;

(F) retrieving from one or more memories, or generating from a 3D model stored in one or more memories, a training template corresponding to a view that is based on the sensor pose obtained on or after the first pose is lost such that the view is a view of the object just before the object pose is no longer tracked; and

(G) deriving a second pose of the object using the training template.

17. The head-mounted display device according to claim 16 ,

wherein (D) includes

predicting the sensor pose of the inertial sensor based on the sensor data sequence using a non-linear filter algorithm.

18. The head-mounted display device according to claim 17 ,

wherein the method further comprises:

obtaining feature positions of the object for the image frame,

wherein (F) includes

retrieving, from one or more memories, the training template containing information of (i) a view, (ii) template features obtained from an image of a 3D model rendered from the view and/or an image of a reference object captured from the view, and (iii) 3D points corresponding to the template features, where the 3D model and the reference object correspond to the object, and the view is based on the sensor pose obtained on or after the first pose is lost, and

wherein (G) includes

deriving the second pose of the object using the feature positions and the training template.

19. The head-mounted display device according to claim 18 ,

wherein obtaining the 3D points in the image frame includes obtaining the feature positions of the object based on the sensor pose.

20. The head-mounted display device according to claim 18 ,

wherein the method further comprises:

retrieving, from one or more memories, a spatial relationship between the camera and the inertial sensor; and

wherein the view is based on (i) the sensor pose obtained on or after the first pose is lost and (ii) the spatial relationship.