Key-frame selection for parallel tracking and mapping

1. A method of selecting a key image from a plurality of images to construct a coordinate system of an augmented reality system, the method comprising:

determining a first image feature in a first one of the plurality of images corresponding to a marker, wherein the first image feature is determined based on a first camera pose of a camera used to capture the first image at a first location;

determining a second image feature in a second one of the plurality of images captured by the camera at a second location, wherein the second image feature is based on a second camera pose of the camera and the second image feature includes a visual match to the first image feature;

determining a reconstructed position of the marker in a three dimensional (3D) space based on a position of the first image feature in the first image, a position of the second image feature in the second image, the first camera pose, and the second camera pose;

determining a reconstruction error based on the determined reconstructed position of the marker and a pre-determined position of the marker in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.

2. A method according to claim 1 , wherein the pre-determined criterion is dynamically adjusted depending on an accuracy of the augmented reality system set by a user of the augmented reality system.

3. A method according to claim 1 , wherein the pre-determined criterion is dynamically adjusted depending on a mean and a variance of the reconstruction error of a feature of a marker.

4. A method according to claim 1 , wherein the reconstruction error is measured in the constructed coordinate system of the augmented reality system.

5. A method according to claim 1 , further comprising removing the marker from a scene after selecting the first image for constructing the coordinate system of the scene.

6. A method according to claim 1 , wherein the first image and the second image are captured by a multi-view camera system comprising a plurality of cameras.

7. A method according to claim 1 , further comprising producing an instruction for a user movement in response to the reconstruction error not satisfying the pre-determined criterion.

8. An apparatus for selecting a key image from a plurality of images to construct a coordinate system of an augmented reality system, the apparatus comprising:

a memory for storing data and computer program;

a processor coupled to the memory for executing the computer program, the computer program comprising instructions for:

determining a first image feature in a first one of the plurality of images corresponding to a marker, wherein the first image feature is determined based on a first camera pose of a camera used to capture the first image at a first location;

determining a second image feature in a second one of the plurality of images captured by the camera at a second location, wherein the second image feature is based on a second pose of the camera and the second image feature includes a visual match to the first image feature;

determining a reconstructed position of the marker in a three dimensional (3D) space based on a position of the first image feature in the first image, a position of the second image feature in the second image, the first camera pose, and the second camera pose;

determining a reconstruction error based on the determined reconstructed position of the marker and a pre-determined position of the marker in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.

9. A non-transitory computer readable medium storing a computer program to cause an apparatus to perform a method for selecting a key image from a plurality of images for constructing a coordinate system of an augmented reality system, the method comprising:

determining a first image feature in a first one of the plurality of images corresponding to a marker, wherein the first image feature is determined based on a first pose of a camera used to capture the first image at a first location;

determining a second image feature in a second one of the plurality of images captured by the camera at a second location, wherein the second image feature is based on a second pose of the camera and the second image feature includes a visual match to the first image feature;

determining a reconstructed position of the marker in a three dimensional (3D) space based on positions of the first image feature in the first image and second image feature in the second image, and the first and second camera pose;

determining a reconstruction error based on the determined reconstructed position of the marker and a pre-determined position of the marker in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.

10. A method of selecting a key image from a plurality of images to construct a coordinate system of an augmented reality system, the method comprising:

determining a first image feature in a first image captured in a first camera pose, wherein the first image feature corresponds to a feature in a three dimensional (3D) space;

determining a second image feature in a second image captured in a second camera pose, wherein the second image feature corresponds to the feature in the three dimensional (3D) space;

determining a reconstructed position of the feature in the three dimensional (3D) space based on a position of the first image feature in the first image, a position of the second image feature in the second image, the first camera pose, and the second camera pose;

determining a reconstruction error based on the determined reconstructed position of the feature and a pre-determined position of the feature in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.

11. A method according to claim 10 , wherein the pre-determined criterion is dynamically adjusted depending on an accuracy of the augmented reality system set by a user of the augmented reality system.

12. A method according to claim 10 , wherein the pre-determined criterion is dynamically adjusted depending on a mean and a variance of the reconstruction error of a feature of a marker.

13. A method according to claim 10 , wherein the reconstruction error is measured in the constructed coordinate system of the augmented reality system.

14. A method according to claim 10 , further comprising removing a marker from a scene after selecting the first image for constructing the coordinate system of the scene.

15. A method according to claim 10 , wherein the first image and the second image are captured by a multi-view camera system comprising a plurality of cameras.

16. A method according to claim 10 , further comprising producing an instruction for a user movement in response to the reconstruction error not satisfying the pre-determined criterion.

17. An apparatus for selecting a key image from a plurality of images to construct a coordinate system of an augmented reality system, the apparatus comprising:

a memory for storing data and computer program;

a processor coupled to the memory for executing the computer program, the computer program comprising instructions for:

determining a first image feature in a first image captured in a first camera pose, wherein the first image feature corresponds to a feature in a three dimensional (3D) space;

determining a second image feature in a second image captured in a second camera pose, wherein the second image feature corresponds to the feature in the three dimensional (3D) space;

determining a reconstructed position of the feature in the three dimensional (3D) space based on a position of the first image feature in the first image, a position of the second image feature in the second image, the first camera pose, and the second camera pose;

determining a reconstruction error based on the determined reconstructed position of the feature and a pre-determined position of the feature in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.

18. A non-transitory computer readable medium storing a computer program to cause an apparatus to perform a method for selecting a key image from a plurality of images to construct a coordinate system of an augmented reality system, the method comprising:

determining a first image feature in a first image captured in a first camera pose, wherein the first image feature corresponds to a feature in a three dimensional (3D) space;

determining a second image feature in a second image captured in a second camera pose, wherein the second image feature corresponds to the feature in the three dimensional (3D) space;

determining a reconstructed position of the feature in the three dimensional (3D) space based on a position of the first image feature in the first image, a position of the second image feature in the second image, the first camera pose, and the second camera pose;

determining a reconstruction error based on the determined reconstructed position of the feature and a pre-determined position of the feature in the 3D space; and

selecting at least one of the first image and second image as the key image for constructing the coordinate system of the augmented reality system in an event that the determined reconstruction error satisfies a pre-determined criterion for constructing the coordinate system.