Object localization system
Fiducial patterns that produce 2D Barker code-like diffraction patterns at a camera sensor are etched or otherwise provided on a cover glass in front of a camera. 2D Barker code kernels, when cross-correlated with the diffraction patterns captured in images by the camera, provide sharp cross-correlation peaks. Misalignment of the cover glass with respect to the camera can be derived by detecting shifts in the location of the detected peaks with respect to calibrated locations. Devices that include multiple cameras behind a cover glass with one or more fiducials on the cover glass in front of each camera are also described. The diffraction patterns caused by the fiducials at the various cameras may be analyzed to detect movement or distortion of the cover glass in multiple degrees of freedom.
1. A system, comprising:
a camera comprising a camera lens and a camera sensor;
a cover glass between an object field and the camera lens, the cover glass comprising a fiducial pattern configured to affect light received through the cover glass from the object field to cause a diffraction pattern in images formed by the camera lens at a surface of the camera sensor; and
one or more processors configured to:
apply a correlation technique to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor;
determine offsets of the cover glass with respect to the camera lens from the located centroid; and
apply the determined offsets to one or more images captured by the camera to account for distortion in the one or more images caused by a corresponding shift in the cover glass with respect to the camera lens during processing of the one or more images.
2. The system as recited in claim 1 , wherein, to determine offsets of the cover glass with respect to the camera lens from the located centroid, the one or more processors are configured to compare a location of the centroid on the camera sensor to a known location on the camera sensor determined during a calibration process.
3. The system as recited in claim 1 , wherein, apply a correlation technique to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor, the one or more processors are configured to apply a correlation kernel corresponding to the diffraction pattern to the at least one image captured by the camera to locate the centroid of the diffraction pattern on the camera sensor.
4. The system as recited in claim 2 , wherein the correlation kernel is a two-dimensional (2D) Barker code.
5. The system as recited in claim 2 , wherein the correlation kernel is a sine-modulated two-dimensional (2D) Barker code.
6. The system as recited in claim 2 , wherein the diffraction pattern is a sine-modulated two-dimensional (2D) Barker code diffraction pattern, wherein the correlation kernel is a 2D Barker code, and wherein the one or more processors are configured to apply a demodulation method to the one or more images to demodulate the sine-modulated Barker code diffraction pattern prior to applying the correlation kernel to the one or more images.
7. The system as recited in claim 2 , wherein the correlation kernel is a circular two-dimensional (2D) Barker code.
8. The system as recited in claim 2 , wherein the correlation kernel is a two-dimensional (2D) random code.
9. The system as recited in claim 1 , wherein, to apply a correlation technique to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor, the one or more processors are configured to:
apply the correlation technique to multiple images captured by the camera to locate the diffraction patterns on the camera sensor;
average the diffraction patterns across the multiple images; and
locate the centroid of the diffraction pattern on the camera sensor from the averaged diffraction patterns.
10. The system as recited in claim 1 , wherein the camera and the cover glass are components of a head-mounted device (HMD).
11. The system as recited in claim 1 ,
wherein the cover glass comprises two or more fiducial patterns configured to affect light received from the object field to cause two or more diffraction patterns in images formed by the camera lens at the surface of the camera sensor; and
wherein, to apply a correlation technique to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor, the one or more processors are configured to:
apply respective correlation kernels corresponding to the diffraction patterns to at least one image captured by the camera to locate centroids of the diffraction patterns on the camera sensor; and
determine offsets of the cover glass with respect to the camera lens from the located centroids.
12. The system as recited in claim 1 ,
wherein the system comprises two or more cameras located behind the cover glass, each camera comprising a camera lens and a camera sensor;
wherein, for each of the two or more cameras, the cover glass comprises one or more fiducial patterns configured to affect light received from the object field to cause a respective one or more diffraction patterns in images formed by the respective camera lens at a surface of the respective camera sensor;
wherein, to apply a correlation technique to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor, the one or more processors are configured to:
apply respective correlation kernels corresponding to the diffraction patterns to images captured by the two or more cameras to locate centroids of the diffraction patterns on the camera sensors; and
determine distortion or shift of the cover glass with respect to the camera lenses from the located centroids.
13. A method, comprising:
receiving light from an object field through a cover glass between the object field and a camera lens, the cover glass including a fiducial pattern;
refracting, by the camera lens, the light received through the cover glass to form an image at a surface of a camera sensor, wherein the fiducial pattern affects the light to cause a diffraction pattern in the image;
capturing, by the camera sensor, one or more images;
applying, by one or more processors, a correlation kernel corresponding to the diffraction pattern to at least one image captured by the camera sensor to locate a centroid of the diffraction pattern on the camera sensor;
determining, by the one or more processors, a shift of the cover glass with respect to the camera lens from the located centroid; and
adjusting processing of one or more images captured by the camera to account for the determined shift in the cover glass with respect to the camera lens.
14. The method as recited in claim 13 , wherein determining the shift of the cover glass with respect to the camera lens from the located centroid comprises comparing a location of the centroid on the camera sensor to a known location on the camera sensor determined during a calibration process.
15. The method as recited in claim 13 , wherein the correlation kernel is one of a two-dimensional (2D) Barker code or a sine-modulated two-dimensional (2D) Barker code.
16. The method as recited in claim 13 , wherein the diffraction pattern is a sine-modulated two-dimensional (2D) Barker code diffraction pattern, wherein the correlation kernel is a 2D Barker code, and wherein the method further comprising applying a demodulation method to the one or more images to demodulate the sine-modulated Barker code diffraction pattern prior to applying the correlation kernel to the one or more images.
17. The method as recited in claim 13 , wherein the correlation kernel is one of a circular two-dimensional (2D) Barker code and a two-dimensional (2D) random code.
18. The method as recited in claim 13 , wherein applying a correlation kernel corresponding to the diffraction pattern to at least one image captured by the camera to locate a centroid of the diffraction pattern on the camera sensor comprises:
applying the correlation kernel to multiple images captured by the camera to locate the diffraction patterns on the camera sensor;
averaging the diffraction patterns across the multiple images; and
locating the centroid of the diffraction pattern on the camera sensor from the averaged diffraction patterns.
19. The method as recited in claim 13 , wherein the cover glass includes two or more fiducial patterns that affect light received from the object field to cause two or more diffraction patterns in images formed by the camera lens at the surface of the camera sensor, the method further comprising:
applying respective correlation kernels corresponding to the diffraction patterns to at least one image captured by the camera to locate centroids of the diffraction patterns on the camera sensor; and
determining shift of the cover glass with respect to the camera lens from the located centroids.
20. The method as recited in claim 13 , wherein there are two or more cameras located behind the cover glass, each camera comprising a camera lens and a camera sensor, wherein, for each of the two or more cameras, the cover glass comprises one or more fiducial patterns configured to affect light received from the object field to cause a respective one or more diffraction patterns in images formed by the respective camera lens at a surface of the respective camera sensor, the method further comprising:
applying respective correlation kernels corresponding to the diffraction patterns to images captured by the two or more cameras to locate centroids of the diffraction patterns on the camera sensors; and
determining distortion or shift of the cover glass with respect to the camera lenses from the located centroids.