Methods for facilitating computer vision application initialization
Embodiments disclosed pertain to systems, method s and apparatus for the initialization of Computer Vision (CV) applications on user devices (UDs) comprising a camera and a display. In some embodiments, an optimal camera trajectory for initialization of a Computer Vision (CV) application may be determined based on an initial camera pose and an estimated pivot distance. For example, the initial camera pose may be estimated based on a first image captured by the camera. Further, the display may be updated in real-time with an indication of a desired movement direction for the camera. In some embodiments, the indication of desired movement direction may be based, in part, on a current camera pose and the optimal trajectory, where the current camera pose may be estimated based on a current image captured by the camera.
1. A method on a user device (UD) comprising a camera and a display, the method comprising:
determining a trajectory of the camera for targetless initialization of a Computer Vision (CV) application based, in part, on an initial camera pose and an estimated pivot distance between the camera and a pivot position, wherein the initial camera pose is estimated based on a targetless first image captured by the camera; and
updating the display in real-time with an indication of a desired movement direction for the camera, the indication of the desired movement direction being based, in part, on a current camera pose and the trajectory of the camera, wherein the current camera pose is estimated based on a current image captured by the camera.
2. The method of claim 1 , wherein the desired movement direction for the camera is obtained based in part, on a comparison of a current trajectory of the camera with the trajectory of the camera, the current trajectory of the camera being determined from a sequence of prior camera poses.
3. The method of claim 1 , further comprising:
updating the display in real-time upon a movement of the camera with one or more additional indications of desired movement directions for the camera, wherein each additional indication of desired movement direction is based, in part, on a corresponding additional camera pose associated with the movement of the camera.
4. The method of claim 1 , wherein the pivot distance is determined as at least one of:
an arbitrary positive value;
an average distance of a subset of observed feature points in the first image;
a value above a threshold.
5. The method of claim 1 , further comprising:
computing a measure of quality of a current trajectory of the camera, the current trajectory of the camera being determined based, in part, on the current camera pose and a sequence of prior camera poses, and wherein the measure of quality is based on a comparison of the current trajectory of the camera with the trajectory of the camera; and
displaying an indication of the measure of quality of the current trajectory of the camera.
6. The method of claim 1 , wherein determining the trajectory of the camera, comprises:
selecting the first image from one or more camera captured images based, in part, on a number of feature points in the one or more camera captured images.
7. The method of claim 6 , further comprising:
providing, upon selection of the first image, an indication of suitability of the first image for initialization by displaying a first virtual object; or
providing an indication of unsuitability for initialization of the one or more captured images by displaying a second virtual object.
8. The method of claim 1 , wherein the indication of the desired movement direction is provided using at least one of:
a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction; or
a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction; or
a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction; or
a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.
9. The method of claim 1 , wherein the indication of the desired movement direction is provided for both sideward translational and pivot initialization motion sequences.
10. The method of claim 1 , wherein the indication of the desired movement direction comprises an indication of past movement and an indication of a preferred future movement direction.
11. The method of claim 1 , wherein the indication of the desired movement direction also indicates rotational movement of the camera.
12. The method of claim 1 , further comprising:
providing at least one of an auditory feedback or a tactile feedback to a user in conjunction with the indication of the desired movement direction.
13. A user device (UD) comprising:
an image sensor to capture a sequence of images;
a display, and
a processor coupled to the image sensor and the display, wherein the processor is configured to:
determine a trajectory of the image sensor for targetless initialization of a Computer Vision (CV) application based on an initial image sensor pose and an estimated pivot distance between the image sensor and a pivot position, wherein the initial image sensor pose is estimated based on a targetless first image captured by the image sensor, and
update the display in real-time with an indication of a desired movement direction for the image sensor, the indication of the desired movement direction being based, in part, on a current image sensor pose and the trajectory of the image sensor, wherein the current image sensor pose is estimated based on a current image captured by the image sensor.
14. The UD of claim 13 , wherein to obtain the desired movement direction for the image sensor, the processor is configured to:
compare the trajectory of the image sensor with a current trajectory of the image sensor, the current trajectory of the image sensor being determined from a sequence of prior image sensor poses.
15. The UD of claim 13 , the processor is further configured to:
update the display in real-time upon a movement of the image sensor with a one or more additional indications of desired movement directions for the image sensor, wherein each additional indication of desired movement direction is based, in part, on a corresponding additional image sensor pose associated with the movement of the image sensor.
16. The UD of claim 13 , wherein the processor is configured to:
determine the pivot distance as at least one of:
an arbitrary positive value;
an average distance of observed feature points in the first image;
a value above a threshold.
17. The UD of claim 13 , wherein the processor is further configured to:
compute a measure of quality of a current trajectory of the image sensor, the current trajectory of the image sensor being determined based, in part, on the current image sensor pose and a sequence of prior image sensor poses, and wherein the measure of quality is based on a comparison of the current trajectory of the image sensor with the trajectory of the image sensor; and
display an indication of the measure of quality of the current trajectory of the image sensor.
18. The UD of claim 13 , wherein to determine a trajectory of the image sensor, the processor is configured to:
select the first image from one or more image sensor captured images based, in part, on a number of feature points in the one or more image sensor captured images.
19. The UD of claim 18 , wherein the processor is further configured to:
provide, upon selection of the first image, an indication of suitability of the first image for initialization by displaying a first virtual object; or
provide an indication of unsuitability of the one or more captured images for initialization by displaying a second virtual object.
20. The UD of claim 13 , wherein the indication of the desired movement direction is provided using at least one of:
a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction; or
a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction; or
a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction; or
a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.
21. The UD of claim 13 , wherein the indication of the desired movement direction is provided for both sideward translational and pivot initialization motion sequences.
22. The UD of claim 13 , wherein the indication of the desired movement direction comprises an indication of past movement and an indication of a preferred future movement direction.
23. The UD of claim 13 , wherein the indication of the desired movement direction also indicates rotational movement of the UD.
24. The UD of claim 13 , wherein the processor is further configured to:
cause the provision of at least one of an auditory feedback or a tactile feedback to a user in conjunction with the indication of the desired movement direction.
25. An apparatus comprising:
image capture means, the image capture means to capture a sequence of images;
display means;
means for determining a trajectory of the image capture means for targetless initialization of a Computer Vision (CV) application based on an initial pose of the image capture means and an estimated pivot distance between the camera and a pivot position, wherein the initial pose of the image capture means is estimated based on a targetless first image captured by the image capture means; and
means for updating the display means in real-time with an indication of a desired movement direction for the image capture means, the indication of desired movement direction being based, in part, on a current pose of the image capture means and the trajectory of the image capture means, wherein the current pose of the image capture means is estimated based on a current image captured by the image capture means.
26. The apparatus of claim 25 , wherein means for updating the display means comprises:
means for obtaining the desired movement direction for the image sensor, wherein the means for obtaining the desired movement direction for the image sensor comprises:
means for comparing the trajectory of the image capture means with a current trajectory of the image capture means, the current trajectory of the image capture means being determined from a sequence of prior image capture means poses.
27. The apparatus of claim 21 , wherein the indication of the desired movement direction is provided using at least one of:
a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction; or
a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction; or
a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction; or
a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.
28. A non-transitory computer-readable medium comprising instructions, which, when executed by a processor, perform a method on a user device (UD) comprising a camera and a display, the method comprising:
determining a trajectory of a camera for targetless initialization of a Computer Vision (CV) application based on an initial camera pose and an estimated pivot distance between the camera and a pivot position, wherein the initial camera pose is estimated based on a targetless first image captured by the camera; and
updating the display in real-time with an indication of a desired movement direction for the camera, the indication of the desired movement direction being based, in part, on a current camera pose and the trajectory of the camera, wherein the current camera pose is estimated based on a current image captured by the camera.
29. The computer-readable medium of claim 28 , wherein the desired movement direction for the camera is obtained based in part, on a comparison of a current trajectory of the camera with the trajectory of the camera, the current trajectory of the camera being determined from a sequence of prior camera poses.
30. The computer-readable medium of claim 28 , wherein the indication of the desired movement direction is provided using at least one of:
a Screen-aligned 2-Dimensional (2D) guidance scheme comprising 2D screen aligned UI elements to provide the indication of the desired movement direction; or
a Screen-aligned 3-Dimensional (3D) guidance scheme comprising 3D screen aligned UI elements to provide the indication of the desired movement direction; or
a World aligned 3D guidance scheme, comprising 3D UI elements aligned to real world objects to provide the indication of the desired movement direction; or
a World aligned 2.5-Dimensional (2.5-D) guidance scheme, comprising UI elements displayed in 2.5-D, the UI elements aligned to real world objects to provide the indication of the desired movement direction.