Direct scale level selection for multilevel feature tracking under motion blur

1. A method for selective motion blur mitigation in a visual tracking system comprising:

accessing a first image generated by an optical sensor of the visual tracking system;

accessing a second image generated by the optical sensor of the visual tracking system, the second image following the first image;

determining a first motion blur level of the first image;

determining a second motion blur level of the second image;

identifying a scale change between the first image and the second image;

determining a first optimal scale level for the first image based on the first motion blur level and the scale change by:

calculating a first matching level based on the first motion blur level,

applying the scale change to the first matching level to generate a scaled matching level of the first image,

identifying a selected scale level based on a maximum level between the scaled matching level of the first image and a second matching level based on the second motion blur level, and

applying the selected scale level to the first optimal scale level for the first image; and

determining a second optimal scale level for the second image based on the second motion blur level and the scale change.

2. The method of claim 1 , further comprising:

downscaling the first image at the first optimal scale level to generate a first downscaled image; and

downscaling the second image at the first optimal scale level to generate a second downscaled image.

3. The method of claim 2 , further comprising:

identifying a first feature in the first downscaled image;

identifying a second feature in the second downscaled image; and

matching the first feature with the second feature.

4. The method of claim 1 , wherein determining the second optimal scale level for the second image comprises:

calculating a second matching level based on the second motion blur level; and

applying the selected scale level to the second optimal scale level for the second image.

5. The method of claim 1 , further comprising:

calculating a second matching level based on the second motion blur level;

determining a base matching level based on a maximum of the first matching level and the second matching level; and

adjusting the base matching level based on the scale change.

6. The method of claim 1 , wherein determining the first motion blur level comprises:

identifying first camera operating parameters of the optical sensor for the first image; and

determining a first motion of the optical sensor for the first image,

wherein determining the second motion blur level comprises:

identifying second camera operating parameters of the optical sensor for the second image; and

determining a second motion of the optical sensor for the second image.

7. The method of claim 6 , wherein determining the first motion of the optical sensor for the first image comprises:

retrieving first inertial sensor data from an inertial sensor of the visual tracking system for the first image; and

determining a first angular velocity of the visual tracking system based on the first inertial sensor data,

wherein the first motion blur level is based on the first camera operating parameters and the first angular velocity of the visual tracking system without analyzing a content of the first image,

wherein determining the second motion of the optical sensor for the second image comprises:

retrieving second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and

determining a second angular velocity of the visual tracking system based on the second inertial sensor data,

wherein the second motion blur level is based on the second camera operating parameters and the second angular velocity of the visual tracking system without analyzing a content of the second image.

8. The method of claim 6 , wherein determining the first motion of the optical sensor for the first image comprises:

accessing first VIO data from a VIO system of the visual tracking system, the first VIO data comprising a first estimated angular velocity of the optical sensor, a first estimated linear velocity of the optical sensor, and locations of feature points in the first image,

wherein the first motion blur level is based on the first camera operating parameters and the first VIO data without analyzing a content of the first image,

wherein the first motion blur in different areas of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the 3D locations of the feature points in the corresponding different areas of the first image with respect to the optical sensor,

wherein determining the first motion of the optical sensor for the first image comprises:

accessing second VIO data from the VIO system of the visual tracking system, the second VIO data comprising a second estimated angular velocity of the optical sensor, a second estimated linear velocity of the optical sensor, and locations of feature points in the second image,

wherein the second motion blur level is based on the second camera operating parameters and the second VIO data without analyzing a content of the second image,

wherein the second motion blur in different areas of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D locations of the feature points in the corresponding different areas of the second image with respect to the optical sensor.

9. The method of claim 6 , wherein the first or the second source camera operating parameters comprise a combination of an exposure time of the optical sensor, a field of view of the optical sensor, an ISO value of the optical sensor, and an image resolution,

wherein the first image comprises a source image, wherein the second image comprises a target image.

10. A computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the apparatus to:

access a first image generated by an optical sensor of a visual tracking system;

access a second image generated by the optical sensor of the visual tracking system, the second image following the first image;

determine a first motion blur level of the first image;

determine a second motion blur level of the second image;

identify a scale change between the first image and the second image;

determine a first optimal scale level for the first image based on the first motion blur level and the scale change by:

calculating a first matching level based on the first motion blur level, applying the scale change to the first matching level to generate a scaled matching level of the first image,

identifying a selected scale level based on a maximum level between the scaled matching level of the first image and a second matching level based on the second motion blur level, and

applying the selected scale level to the first optimal scale level for the first image; and

determine a second optimal scale level for the second image based on the second motion blur level and the scale change.

11. The computing apparatus of claim 10 , wherein the instructions further configure the apparatus to:

downscaling the first image at the first optimal scale level to generate a first downscaled image; and

downscaling the second image at the first optimal scale level to generate a second downscaled image.

12. The computing apparatus of claim 11 , wherein the instructions further configure the apparatus to:

identify a first feature in the first downscaled image;

identify a second feature in the second downscaled image; and

match the first feature with the second feature.

13. The computing apparatus of claim 10 , wherein determining the second optimal scale level for the second image comprises:

calculate a second matching level based on the second motion blur level; and

apply the selected scale level to the second optimal scale level for the second image.

14. The computing apparatus of claim 10 , wherein the instructions further configure the apparatus to:

calculate a second matching level based on the second motion blur level;

determine a base matching level based on a maximum of the first matching level and the second matching level; and

adjust the base matching level based on the scale change.

15. The computing apparatus of claim 10 , wherein determining the first motion blur level comprises:

identify first camera operating parameters of the optical sensor for the first image; and

determine a first motion of the optical sensor for the first image,

wherein determining the second motion blur level comprises:

identify second camera operating parameters of the optical sensor for the second image; and

determine a second motion of the optical sensor for the second image.

16. The computing apparatus of claim 15 , wherein determining the first motion of the optical sensor for the first image comprises:

retrieve first inertial sensor data from an inertial sensor of the visual tracking system for the first image; and

determine a first angular velocity of the visual tracking system based on the first inertial sensor data,

wherein the first motion blur level is based on the first camera operating parameters and the first angular velocity of the visual tracking system without analyzing a content of the first image,

wherein determining the second motion of the optical sensor for the second image comprises:

retrieve second inertial sensor data from the inertial sensor of the visual tracking system for the second image; and

determine a second angular velocity of the visual tracking system based on the second inertial sensor data,

wherein the second motion blur level is based on the second camera operating parameters and the second angular velocity of the visual tracking system without analyzing a content of the second image.

17. The computing apparatus of claim 15 , wherein determining the first motion of the optical sensor for the first image comprises:

access first VIO data from a VIO system of the visual tracking system, the first VIO data comprising a first estimated angular velocity of the optical sensor, a first estimated linear velocity of the optical sensor, and locations of feature points in the first image,

wherein the first motion blur level is based on the first camera operating parameters and the first VIO data without analyzing a content of the first image,

wherein the first motion blur in different areas of the first image is based on the first estimated angular velocity of the optical sensor, the first estimated linear velocity of the optical sensor, and the 3D locations of the feature points in the corresponding different areas of the first image with respect to the optical sensor,

wherein determining the first motion of the optical sensor for the first image comprises:

access second VIO data from the VIO system of the visual tracking system, the second VIO data comprising a second estimated angular velocity of the optical sensor, a second estimated linear velocity of the optical sensor, and locations of feature points in the second image,

wherein the second motion blur level is based on the second camera operating parameters and the second VIO data without analyzing a content of the second image,

wherein the second motion blur in different areas of the second image is based on the second estimated angular velocity of the optical sensor, the second estimated linear velocity of the optical sensor, and the 3D locations of the feature points in the corresponding different areas of the second image with respect to the optical sensor.

18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

access a first image generated by an optical sensor of a visual tracking system;

access a second image generated by the optical sensor of the visual tracking system, the second image following the first image;

determine a first motion blur level of the first image;

determine a second motion blur level of the second image;

identify a scale change between the first image and the second image;

determine a first optimal scale level for the first image based on the first motion blur level and the scale change by:

calculating a first matching level based on the first motion blur level,

applying the scale change to the first matching level to generate a scaled matching level of the first image,

identifying a selected scale level based on a maximum level between the scaled matching level of the first image and a second matching level based on the second motion blur level, and

applying the selected scale level to the first optimal scale level for the first image; and

determine a second optimal scale level for the second image based on the second motion blur level and the scale change.