Active dual pixel stereo system for depth extraction
A miniaturized active dual pixel stereo system and method for close range depth extraction includes a projector adapted to project a locally distinct projected pattern onto an image of a scene and a dual pixel sensor including a dual pixel sensor array that generates respective displaced images of the scene. A three-dimensional image is generated from the displaced images of the scene by projecting the locally distinct projected pattern onto the image of the scene, capturing the respective displaced images of the scene using the dual pixel sensor, generating disparity images from the respective displaced images of the scene, determining depth to each pixel of the disparity images, and generating the three-dimensional image from the determined depth to each pixel. A three-dimensional image of a user's hands generated by the active dual pixel stereo system may be processed by gesture recognition software to provide an input to an electronic eyewear device.
1 . An active dual pixel stereo system for depth extraction, comprising:
a projector adapted to project a locally distinct projected pattern onto an image of a scene, wherein the projected pattern comprises at least one of a visible light pattern or an infrared light pattern;
a dual pixel sensor comprising a dual pixel RGB-infrared sensor array that generates respective infrared and RGB displaced images of the scene by splitting each pixel such that a first aperture of each pixel integrates light over a second aperture of each pixel and the second aperture of each pixel integrates light over the first aperture of each pixel whereby the first and second apertures of each pixel together form a stereo pair exhibiting disparity between two views of each pixel based on a distance from the dual pixel sensor;
a dual focal lens and a dual band-pass filter that selectively pass infrared and visible images to the dual pixel sensor to generate the respective infrared and RGB displaced images of the scene;
a memory that stores instructions; and
at least one processor that executes the instructions to create a three-dimensional image from the infrared and RGB displaced images of the scene by performing operations including:
projecting the locally distinct projected pattern onto the image of the scene;
capturing the respective infrared and RGB displaced images of the scene using the dual pixel sensor;
measuring displacements, using a feature matching algorithm, between respective feature points in the respective infrared and RGB displaced images to generate disparity images from the respective infrared and RGB displaced images of the scene;
determining a depth to each pixel of the disparity images by comparing values at each feature pixel in the disparity images to compute a horizontal distance value and converting the horizontal distance value into a depth value using a disparity map table; and
generating the three-dimensional image from the depth values to each pixel.
2 . An active dual pixel stereo system as in claim 1 , wherein the three-dimensional image comprises an image of a user's hands, further comprising gesture recognition software that is executed by the at least one processor to process the three-dimensional image of the user's hands to generate an input to an electronic eyewear device.
3 . An active dual pixel stereo system as in claim 1 , wherein the dual pixel sensor is sensitive to a wavelength of the projector.
4 . A method of generating three-dimensional images of a scene, comprising:
projecting a locally distinct projected pattern onto an image of the scene, wherein the projected pattern comprises at least one of a visible light pattern or an infrared light pattern;
capturing respective infrared and RGB displaced images of the scene through a dual focal lens and a dual band-pass filter that selectively pass infrared and visible images to a dual pixel sensor comprising a dual pixel RGB-infrared sensor array that generates respective infrared and RGB displaced images of the scene by splitting each pixel such that a first aperture of each pixel integrates light over a second aperture of each pixel and the second aperture of each pixel integrates light over the first aperture of each pixel whereby the first and second apertures of each pixel together form a stereo pair exhibiting disparity between two views of each pixel based on a distance from the dual pixel sensor;
measuring displacements, using a feature matching algorithm, between respective feature points in the respective infrared and RGB displaced images to generate disparity images from the respective infrared and RGB displaced images of the scene;
determining a depth to each pixel of the disparity images by comparing values at each feature pixel in the disparity images to compute a horizontal distance value and converting the horizontal distance value into a depth value using a disparity map table; and
generating the three-dimensional image from the depth values to each pixel.
5 . The method of claim 4 , wherein the three-dimensional image comprises an image of a user's hands, further comprising processing the three-dimensional image of the user's hands to generate an input to an electronic eyewear device.
6 . A non-transitory computer-readable storage medium that stores instructions that when executed by at least one processor cause the at least one processor to generate three-dimensional images of a scene by performing operations comprising:
projecting a locally distinct projected pattern onto an image of the scene, wherein the projected pattern comprises at least one of a visible light pattern or an infrared light pattern;
capturing respective infrared and RGB displaced images of the scene through a dual focal lens and a dual band-pass filter that selectively pass infrared and visible images to a dual pixel sensor comprising a dual pixel RGB-infrared sensor array that generates respective infrared and RGB displaced images of the scene by splitting each pixel such that a first aperture of each pixel integrates light over a second aperture of each pixel and the second aperture of each pixel integrates light over the first aperture of each pixel whereby the first and second apertures of each pixel together form a stereo pair exhibiting disparity between two views of each pixel based on a distance from the dual pixel sensor;
measuring displacements, using a feature matching algorithm, between respective feature points in the respective infrared and RGB displaced images to generate disparity images from the respective infrared and RGB displaced images of the scene;
determining a depth to each pixel of the disparity images by comparing values at each feature pixel in the disparity images to compute a horizontal distance value and converting the horizontal distance value into a depth value using a disparity map table; and
generating the three-dimensional image from the depth values to each pixel.
7 . The computer-readable storage medium of claim 6 , wherein the three-dimensional image comprises an image of a user's hands, further comprising instructions that when executed by the at least one processor further cause the at least one processor to process the three-dimensional image of the user's hands to generate an input to an electronic eyewear device.