Misaligned vantage point mitigation for computer stereo vision
Disclosed are systems, methods, and non-transitory computer-readable media for misaligned vantage point mitigation for computer stereo vision. A misaligned vantage point mitigation system determines whether vantage points of the optical sensors are misaligned from an expected vantage point and, if so, determines an adjustment variable to mitigate the misalignment based on the location of matching features identified in images captured by both optical sensors.
1. A method comprising:
accessing a first image generated by a first optical sensor of a head-worn device, and a second image generated by a second optical sensor of the head-worn device, the first optical sensor and the second optical sensor having at least partially overlapping fields of view;
identifying a first set of features at a horizontal level of the first image;
identifying a second set of features at the horizontal level of the second image;
comparing the first set of features to the second set of features, yielding a comparison;
detecting based on the comparison, an image feature depicted within both the first set of features of the first image and the second image;
determining an expected location of the image feature in the second image based on the first image and expected vantage points of the first optical sensor and the second optical sensor, the expected location of the image feature in the second image corresponding to an alignment of the first optical sensor and the second optical sensor;
determining a variance between the expected location of the image feature in the second image and an actual location of first feature in the second image, the variance indicating that the first optical sensor and the second optical sensor are misaligned due to bending of the head-worn device;
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance to mitigate the misalignment of the first optical sensor and the second optical sensor due to bending of the head-worn device; and
causing display of a presentation of Augmented-Reality (AR) content based on the adjusted expected vantage points of the first optical sensor and the second optical sensor, the presentation of the AR content based on a location of a display of a real-world object.
2. The method of claim 1 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
comparing the variance with a threshold variance;
determining that the variance meets or exceeds the threshold variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor in response to determining that the variance meets of exceeds the threshold variance.
3. The method of claim 2 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
calculating an adjustment variable based on the variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the adjustment variable.
4. The method of claim 1 , wherein the variance comprises a numeric value that indicates a magnitude of an offset of the first feature in the first image and the second image, and a direction of the variance.
5. A system comprising:
one or more computer processors; and
one or more computer readable mediums storing instructions that, when executed by the one or more computer processors, causes the system to perform operations comprising:
accessing a first image generated by a first optical sensor of a head-worn device, and a second image generated by a second optical sensor of the head-worn device, the first optical sensor and the second optical sensor having at least partially overlapping fields of view;
identifying a first set of features at a horizontal level of the first image;
identifying a second set of features at the horizontal level of the second image;
comparing the first set of features to the second set of features, yielding a comparison;
detecting based on the comparison, an image feature depicted within both the first set of features of the first image and the second image;
determining an expected location of the image feature in the second image based on the first image and expected vantage points of the first optical sensor and the second optical sensor, the expected location of the image feature in the second image corresponding to an alignment of the first optical sensor and the second optical sensor,
determining a variance between the expected location of the image feature in the second image and an actual location of first feature in the second image, the variance indicating that the first optical sensor and the second optical sensor are misaligned due to bending of the head-worn device;
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance to mitigate the misalignment of the first optical sensor and the second optical sensor due to bending of the head-worn device; and
causing display of a presentation of Augmented-Reality (AR) content based on the adjusted expected vantage points of the first optical sensor and the second optical sensor, the presentation of the AR content based on a location of a display of a real-world object.
6. The system of claim 5 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
comparing the variance with a threshold variance;
determining that the variance meets or exceeds the threshold variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor in response to determining that the variance meets of exceeds the threshold variance.
7. The system of claim 6 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
calculating an adjustment variable based on the variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the adjustment variable.
8. The system of claim 5 , wherein the variance comprises a numeric value that indicates a magnitude of an offset of the first feature in the first image and the second image, and a direction of the variance.
9. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of one or more computing devices, cause the one or more computing devices to perform operations comprising:
accessing a first image generated by a first optical sensor of a head-worn device, and a second image generated by a second optical sensor of the head-worn device, the first optical sensor and the second optical sensor having at least partially overlapping fields of view;
identifying a first set of features at a horizontal level of the first image;
identifying a second set of features at the horizontal level of the second image;
comparing the first set of features to the second set of features, yielding a comparison;
detecting based on the comparison, an image feature depicted within both the first set of features of the first image and the second image;
determining an expected location of the image feature in the second image based on the first image and expected vantage points of the first optical sensor and the second optical sensor, the expected location of the image feature in the second image corresponding to an alignment of the first optical sensor and the second optical sensor;
determining a variance between the expected location of the image feature in the second image and an actual location of first feature in the second image, the variance indicating that the first optical sensor and the second optical sensor are misaligned due to bending of the head-worn device;
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance to mitigate the misalignment of the first optical sensor and the second optical sensor due to bending of the head-worn device; and
causing display of a presentation of Augmented-Reality (AR) content based on the adjusted expected vantage points of the first optical sensor and the second optical sensor, the presentation of the AR content based on a location of a display of a real-world object.
10. The non-transitory machine-readable storage medium of claim 9 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
comparing the variance with a threshold variance;
determining that the variance meets or exceeds the threshold variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor in response to determining that the variance meets of exceeds the threshold variance.
11. The non-transitory machine-readable storage medium of claim 10 , wherein adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the variance further comprises:
calculating an adjustment variable based on the variance; and
adjusting the expected vantage points of the first optical sensor and the second optical sensor based on the adjustment variable.
12. The non-transitory machine-readable storage medium of claim 9 , wherein the variance comprises a numeric value that indicates a magnitude of an offset of the first feature in the first image and the second image, and a direction of the variance.
13. The method of claim 1 , wherein the first image and the second image are generated during a rolling shutter image capture.
14. The method of claim 13 , further comprising:
assessing angular rates and orientations of the head-worn device during the rolling shutter image capture; and
determining the expected location of the image feature in the second image based on the first image, the expected vantage points of the first optical sensor and the second optical sensor, and the angular rates and orientations of the head-worn device during the rolling shutter image capture.
15. The method of claim 14 , wherein the angular rates and orientations are measured by an Inertia Measurement Unit (IMU) mounted on the head-worn device.