Measurement based on point selection
Various implementations disclosed herein include devices, systems, and methods that provide measurement techniques for electronic devices such as optical see-through head mounted devices. In some implementations, a line of sight technique is used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, different measurement point identification techniques are automatically selected and used to identify a 3D position of a measurement point to enable measurement of an object in a 3D environment. In some implementations, a 3D position of a measurement point is identified to enable measurement of an object in a 3D environment, where the measurement point is identified by selecting from multiple candidates that are determined using different measurement point selection techniques.
1. A method comprising:
at a processor:
detecting a finger in a 3D environment;
determining, based on an image of the 3D environment, a 3D position of a measurement point based on whether the finger is at least a first threshold distance away from the measurement point and whether the finger is within a second threshold distance to the measurement point, wherein the first threshold distance is larger than the second threshold distance, wherein if the finger is determined to be at least the first threshold distance away from the measurement point, the 3D position of the measurement point is determined based on a 3D position of the finger and a 3D position of an eye of a user, and if the finger is determined to be within the second threshold distance to the measurement point, the 3D position of the measurement point is determined based on the 3D position of the finger; and
providing a measurement based on the 3D position of the measurement point.
2. The method of claim 1 , wherein detecting the finger comprises detecting the finger in an image of the 3D environment based on color, shape, or connection to an arm or wrist.
3. The method of claim 1 , further comprising determining whether the finger is at least the first threshold distance away from the measurement point and whether the finger is within the second threshold distance to the measurement point (i) using depth around a fingertip of the finger in the 3D environment and a depth of the fingertip, (ii) evaluating an appearance or a positioning of the finger relative to a mobile electronic device, (iii) detecting a shadow of the finger, (iv) evaluating skin deformation or color modification of the finger when touching, or (v) estimating a distance from a surface of interest to the finger using depth estimation and gaze tracking.
4. The method of claim 1 , wherein determining the 3D position of the measurement point further comprises identifying an object in the 3D environment adjacent to a fingertip of the finger.
5. The method of claim 1 , wherein the 3D position of the measurement point is determined based on the 3D position of the finger and the 3D position of the eye of the user by:
extending a line from the 3D position of the eye through the 3D position of the finger; and
determining an intersection of the line and an object in the 3D environment.
6. The method of claim 1 , wherein the measurement is a distance to the measurement point, a distance between the measurement point and a second measurement point, a surface area identified based on the measurement point, or a volume identified based on the measurement point.
7. The method of claim 6 , further comprising determining a set of measurement points, wherein providing the measurement is further based on the set of measurement points.
8. A system comprising:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the system to:
detect a finger in a 3D environment;
determine, based on an image of the 3D environment, a 3D position of a measurement point based on whether the finger is at least a first threshold distance away from the measurement point and whether the finger is within a second threshold distance to the measurement point, wherein the first threshold distance is larger than the second threshold distance, wherein if the finger is determined to be at least the first threshold distance away from the measurement point, the 3D position of the measurement point is determined based on a 3D position of the finger and a 3D position of an eye of a user, and if the finger is determined to be within the second threshold distance to the measurement point, the 3D position of the measurement point is determined based on the 3D position of the finger; and
provide a measurement based on the 3D position of the measurement point.
9. The system of claim 8 , wherein the instructions, when executed, further cause the system to detect the finger in the 3D environment based on color, shape, or connection to an arm or wrist.
10. The system of claim 8 , wherein the instructions, when executed, further cause the system to determine whether the finger is at least the first threshold distance away from the measurement point and whether the finger is within the second threshold distance to the measurement point based on whether the finger is touching an object in the 3D environment or whether the finger is aligned with the object in the 3D environment.
11. The system of claim 8 , wherein the instructions, when executed, further cause the system to determine the 3D position of the eye of the user based on whether the eye of the user is open, is a dominant eye, or is a preselected eye.
12. The system of claim 8 , wherein the instructions, when executed, further cause the system to determine the 3D position of the measurement point based on the 3D position of the finger and the 3D position of the eye of the user by:
extending a line from the 3D position of the eye through the 3D position of the finger; and
determining an intersection of the line and an object in the 3D environment.
13. The system of claim 8 , wherein the measurement is a distance to the measurement point, a distance between the measurement point and a second measurement point, a surface area identified based on the measurement point, or a volume identified based on the measurement point.
14. The system of claim 13 , wherein a set of a plurality of measurement points is determined, and the set of measurement points is used to provide the measurement.
15. The system of claim 8 , wherein the instructions, when executed, further cause the system to determine whether the finger is at least the first threshold distance away from the measurement point and whether the finger is within the second threshold distance to the measurement point (i) using depth around a fingertip of the finger in the 3D environment and a depth of the fingertip, (ii) evaluating an appearance or a positioning of the finger relative to a mobile electronic device, (iii) detecting a shadow of the finger, (iv) evaluating skin deformation or color modification of the finger when touching, or (v) estimating a distance from a surface of interest to the finger using depth estimation and gaze tracking.
16. A computer-readable storage medium storing instructions that, when executed cause performance of:
detecting a finger in a 3D environment;
determining, based on an image of the 3D environment, whether the finger is at least a first threshold distance away from a measurement point or the finger is within a second threshold distance to the measurement point, wherein the first threshold distance is larger than the second threshold distance;
based on determining that the finger is at least the first threshold distance away from the measurement point, determining a 3D position of the measurement point based on a 3D position of the finger and a 3D position of an eye of a user;
based on determining that the finger is within the second threshold distance to the measurement point, determining the 3D position of the measurement point based on the 3D position of the finger; and
providing a measurement based on the 3D position of the measurement point.
17. The storage medium of claim 16 , wherein detecting the finger in the 3D environment is based on color, shape, or connection to an arm or wrist.
18. The storage medium of claim 16 , further storing instructions that, when executed, further cause determining whether the finger is touching an object in the 3D environment or whether the finger is aligned with the object in the 3D environment.
19. The storage medium of claim 16 , further comprising instructions that, when executed, further cause determining the 3D position of the eye of the user based on whether the eye of the user is open, is a dominant eye, or is a preselected eye.
20. The storage medium of claim 16 , further comprising instructions that, when executed, further cause determining the 3D position of the measurement point based on the 3D position of the finger and the 3D position of the eye of the user by:
extending a line from the 3D position of the eye through the 3D position of the finger; and
determining an intersection of the line and an object in the 3D environment.