Dual camera system for real-time depth map generation

1. A method of real-time depth mapping in a dual camera system, comprising:

receiving an auto focus image from an auto focus camera module;

converting the auto focus image to monochrome;

rectifying the converted auto focus image;

receiving a fixed focus image from a plenoptic camera module having lens that cover two by two pixels and outputting a plurality of plenoptic sub-images;

extracting four plenoptic sub-images from the plurality of plenoptic sub images;

averaging the extracted plenoptic sub-images;

rectifying the averaged plenoptic sub-images;

block matching the rectified auto focus image and the rectified plenoptic sub-images along epipolar lines, outputting an epipolar block match;

block matching the rectified plenoptic sub-images along a vertical direction, outputting vertical plenoptic block match;

block matching the rectified plenoptic sub-images along a horizontal direction, outputting horizontal plenoptic block match;

determining a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;

determining a plenoptic disparity map of the extracted plenoptic sub-images based on the composite block matching error,

converting the plenoptic disparity map to a plenoptic depth map;

converting the plenoptic depth map to an initial stereo disparity map;

searching for a guided disparity map based on the initial stereo disparity map and the epipolar block match; and

converting the guided disparity map to a guided depth map.

2. The method of real-time depth mapping in the dual camera system of claim 1 , further comprising determining a plenoptic confidence map of the extracted plenoptic sub-images based on the composite block matching error.

3. The method of real-time depth mapping in the dual camera system of claim 1 , further comprising receiving a color image from the auto focus camera module.

4. The method of real-time depth mapping in the dual camera system of claim 3 , further comprising smoothing the guided disparity map.

5. The method of real-time depth mapping in the dual camera system of claim 4 , further comprising converting the smoothed guided disparity map to a smoothed depth map.

6. The method of real-time depth mapping in the dual camera system of claim 1 , wherein the block matching is performed in the horizontal direction.

7. The method of real-time depth mapping in the dual camera system of claim 6 , wherein the block matching is performed in the vertical direction.

8. The method of real-time depth mapping in the dual camera system of claim 7 , wherein the block matching error is a composite block matching error based on the block matching performed in the vertical direction and the horizontal direction.

9. The method of real-time depth mapping in the dual camera system of claim 8 , further comprising:

determining a disparity map of the rectified plenoptic sub-images based on the horizontal block matching error and the vertical block matching error; and

converting the disparity map to a depth map.

10. A dual camera system having real-time depth map generation, comprising:

an auto focus camera module outputting an auto focus image;

a monochromatic plenoptic camera module, wherein a plenoptic lens cover two by two pixels, outputting a plurality of plenoptic sub-images, wherein the plenoptic camera module is fixed focus;

a processor coupled to the auto focus camera module and the plenoptic camera module, wherein the processor;

converts the auto focus image to monochrome;

rectifies the converted auto focus image;

extracts four plenoptic sub-images from the plenoptic camera module;

averages the four plenoptic sub-images from the plurality of plenoptic sub-images;

rectifies the averaged four plenoptic sub-images; and

block matches the rectified auto focus image and the rectified four plenoptic sub-images along epipolar lines.

11. The dual camera system having real-time depth map generation of claim 10 , wherein the processor further

extracts four plenoptic sub-images from the plurality of plenoptic sub-images;

block matches the four plenoptic sub-images along a vertical direction, outputting a vertical plenoptic block match;

block matches the four plenoptic sub-images along a horizontal direction, outputting a horizontal plenoptic block match;

determines a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;

determines a plenoptic disparity map of the four plenoptic sub-images based on the composite block matching error;

determines a plenoptic confidence map of the four plenoptic sub-images based on the composite block matching error, and

converts the plenoptic disparity map to a plenoptic depth map.

12. The dual camera system having real-time depth map generation of claim 11 , wherein the processor further

converts the plenoptic depth map to an initial stereo disparity map;

searches for a guided disparity map based on the initial stereo disparity map;

smoothes the guided disparity map; and

converts the guided disparity map to a guided depth map.

13. The dual camera system having real-time depth map generation of claim 12 , wherein the processor further smoothes the guided depth map.

14. The dual camera system having real-time depth map generation of claim 13 , wherein the guided disparity map is determined from a displacement having a smallest matching error based on the block matches.

15. A dual camera stereo system having real-time depth map generation, comprising:

an auto focus camera module outputting an auto focus image;

a monochromatic plenoptic camera module, wherein a plenoptic lens cover two by two pixels and outputs a plurality of plenoptic sub-images and wherein the plenoptic camera module is fixed focus;

a processor coupled to the auto focus camera module and the plenoptic camera module, wherein the processor;

extracts four plenoptic sub-images from the plurality of plenoptic sub-images;

block matches the four plenoptic sub-images along a vertical direction, outputting a vertical plenoptic block match;

block matches the four plenoptic sub-images along a horizontal direction, outputting a horizontal plenoptic block match;

determines a composite block matching error of the vertical plenoptic block match and the horizontal plenoptic block match;

determines a plenoptic disparity map of the four plenoptic sub-images based on the composite block matching error;

determines a plenoptic confidence map of the four plenoptic sub-images based on the composite block matching error; and

converts the plenoptic disparity map to a plenoptic depth map.

16. The dual camera stereo system having real-time depth map generation of claim 15 , wherein the processor further:

converts the auto focus image to monochrome;

rectifies the converted auto focus image;

averages the four plenoptic sub-images;

rectifies the averaged four plenoptic sub-images;

block matches the rectified auto focus image and the rectified four plenoptic along epipolar lines;

converts the plenoptic depth map to an initial stereo disparity map;

searches the auto focus image based on the initial stereo disparity map in a guided disparity search outputting a guided disparity map; and

converts the guided disparity map to a guided depth map.

17. The dual camera stereo system having real-time depth map generation of claim 16 , wherein the processor further smoothes the guided disparity map.

18. The dual camera stereo system having real-time depth map generation of claim 17 , wherein the processor further converts the smoothed guided disparity map to a smoothed depth map.

19. The dual camera stereo system having real-time depth map generation of claim 16 , wherein the guided disparity map is determined from a displacement having a smallest matching error based on the block matches.