System and method for real time 2D to 3D conversion of video in a digital camera
View Patent ↗Embodiments are directed towards enabling digital cameras to digitally process captured two dimensional image sequences at a real time video rate, to convert the two dimensional image sequences into stereoscopic three dimensional image sequences. In one embodiment, using a pipelining architecture, various statistics are obtained for a captured two dimensional sequence. The statistics are used to estimate depth within each frame image within the sequence, in real time. Using the depth data a disparity map is generated that is provided to a warping component to generate a second perspective of the two dimensional image. The two images for the frame provide a three dimensional perspective for the frame within the sequence. Together the two perspective images for the frame are provided to a video encoder component to encode the stereoscopic three dimensional frame for the sequence.
1. A processor based method, comprising:
capturing a two dimensional image frame sequence with an image sensor; and
performing a plurality of actions at a real time video rate with at least one integrated circuit having a plurality of components thereon operating in a pipeline architecture, including:
receiving a captured two dimensional image frame in the two dimensional image frame sequence, the received frame representing a first view perspective;
dividing the received frame into a plurality of windows;
determining auto focus, auto exposure, and auto white balance statistics for the received frame, the determination of auto focus statistics comprising calculating, for each window of the received frame, a maximal derivative by a maximal pixel value, a minimal pixel value, and an average pixel value of the window;
for each window, (i) determining a lens blurring kernel by using the calculated maximal derivative for the window and (ii), in response to an edge being present in the window, convolving the edge and the determined lens blurring kernel;
correcting lens distortion by applying a transformation to the received frame;
generating estimated depths for the transformed frame based at least in part on the maximal derivatives of the plurality of windows;
generating a disparity map by using the estimated depths for the transformed frame; and
using the disparity map to generate from the transformed frame a second view perspective for the received frame, the received frame and the generated second view forming a stereoscopic image pair providing three dimensional perception.
2. The method of claim 1 , wherein generating the estimated depths comprises:
determining whether the maximal derivatives validly estimate the depth for each of the plurality of windows based at least on a comparison of the maximal pixel value for each window being unsaturated, the minimal pixel value being above an intrinsic noise level, the average pixel value being not close to the maximal or minimal pixel values by a threshold value, and the maximal derivative being above another threshold; and
when the maximal derivative does not validly estimate the depth for the window:
scanning each window for the given frame to identify connected invalid windows;
determining a border of windows surrounding the connected invalid windows; and
replacing the maximal derivatives for each of the surrounding connected invalid windows with values obtained from the border of windows.
3. The method of claim 1 , wherein using the disparity map to generate from the transformed frame the second view perspective for the received frame, further comprises:
for each window:
using a disparity value from the disparity map as a disparity of a pixel at a center of the window; and
interpolating disparity values for each other pixel within the window using the disparity at the center; and
warping the transformed frame using the disparity values for each of the pixels for the transformed frame to generate the second view perspective.
4. The method of claim 1 , wherein the plurality of components include at least a statistics component used to determine the auto focus statistics and generate the estimated depths, and a warping component used to correct lens distortion within the first view perspective.
5. The method of claim 1 , wherein generating estimated depths comprises applying a spatial-temporal filter to estimated depths for the transformed frame using depth estimates from at least two previous frames within the two dimensional image frame sequence.
6. The method of claim 1 , wherein performing the plurality of actions comprises:
encoding the three dimensional image frame using at least one three dimensional video packing format, including at least one of a field/frame packing, a planar or interleaved mechanism, a top-bottom mechanism, or a side-by-side mechanism.
7. An image system, comprising:
an image sensor configured to receive a two dimensional image frame sequence; and
at least one integrated circuit having a plurality of components thereon operating in a pipeline architecture to perform a plurality of actions at a real time video rate, including:
receiving a received two dimensional image frame in the two dimensional image frame sequence, the received frame representing a first view perspective;
dividing the received frame into a plurality of windows;
determining auto focus, auto exposure, and auto white balance statistics for the received frame, the determination of auto focus statistics comprising calculating, for each window of the received frame, a maximal derivative by using a maximal pixel value, a minimal pixel value, and an average pixel value of the window;
for each window, (i) determining a lens blurring kernel by using the calculated maximal derivative for the window and (ii), in response to an edge being present in the window, convolving the edge and the determined lens blurring kernel;
correcting lens distortion by applying a transformation to the received frame;
generating estimated depths for the transformed frame based at least in part on the maximal derivatives of the plurality of windows;
generating a disparity map by using the estimated depths for the transformed frame; and
using the disparity map to generate from the transformed frame a second view perspective for the received frame, the received frame and the generated second view perspective forming a stereoscopic image pair providing three dimensional perception.
8. The image system of claim 7 , wherein generating the estimated depths comprises:
determining whether the maximal derivatives validly estimate the depth for each of the plurality of windows based at least on a comparison of the maximal pixel value for each window being unsaturated, the minimal pixel value being above an intrinsic noise level, the average pixel value being not close to the maximal or minimal pixel values by a threshold value, and the maximal derivative value is being above another threshold; and
when the maximal derivative does not validly estimate the depth for the window:
scanning each window for the given frame to identify connected invalid windows;
determining a border of windows surrounding the connected invalid windows; and
replacing the maximal derivatives for each of the surrounding connected invalid windows with values obtained from the border of windows.
9. The image system of claim 7 , wherein using the disparity map to generate from the transformed frame the second view perspective for the received frame, further comprises:
for each window:
using a disparity value from the disparity map as a disparity of a pixel at a center of the window; and
interpolating disparity values for each other pixel within the window using the disparity at the center; and
warping the transformed frame using the disparity values for each of the pixels for the transformed frame to generate the second view perspective.
10. The image system of claim 7 , wherein the plurality of components include at least a statistics component used to determine the auto focus statistics and generate the estimated depths, and a warping component used to correct lens distortion within the first view perspective.
11. The image system of claim 7 , wherein generating estimated depths comprises applying a spatial-temporal filter to estimated depths for the transformed frame using depth estimates from at least two previous frames within the two dimensional image frame sequence.
12. The image system of claim 7 , wherein to perform the plurality of actions comprises:
encoding the three dimensional image frame using at least one three dimensional video packing format, including at least one of a field/frame packing, a planar or interleaved mechanism, a top-bottom mechanism, or a side-by-side mechanism.
13. A non-transitory computer readable storage medium having stored thereon instructions that, when executed, enables actions comprising:
performing a plurality of actions at a real time video rate by operating in a pipeline architecture, including:
receiving a captured two dimensional image frame in a two dimensional image frame sequence, the received frame representing a first view perspective;
dividing the received frame into a plurality of windows;
determining auto focus, auto exposure, and auto white balance statistics for the received frame, the determination of auto focus statistics comprising calculating, for each window of the received frame, a maximal derivative by a maximal pixel value, a minimal pixel value, and an average pixel value of the window;
for each window, (i) determining a lens blurring kernel by using the calculated maximal derivative for the window and (ii), in response to an edge being present in the window, convolving the edge and the determined lens blurring kernel;
correcting lens distortion by applying a transformation to the received frame;
generating estimated depths for the transformed frame based at least in part on the maximal derivatives of the plurality of windows;
generating a disparity map by using the estimated depths for the transformed frame; and
using the disparity map to generate from the transformed frame a second view perspective for the received frame, the received frame and the generated second view forming a stereoscopic image pair providing three dimensional perception.
14. The non-transitory computer readable storage medium of claim 13 , wherein generating the estimated depths comprises:
determining whether the maximal derivatives validly estimate the depth for each of the plurality of windows based at least on a comparison of the maximal pixel value for each window being unsaturated, the minimal pixel value being above an intrinsic noise level, the average pixel value being not close to the maximal or minimal pixel values by a threshold value, and the maximal derivative being above another threshold; and
when the maximal derivative does not validly estimate the depth for the window:
scanning each window for the given frame to identify connected invalid windows;
determining a border of windows surrounding the connected invalid windows; and
replacing the maximal derivatives for each of the surrounding connected invalid windows with values obtained from the border of windows.
15. The non-transitory computer readable storage medium of claim 13 , wherein using the disparity map to generate from the transformed frame the second view perspective for the received frame, further comprises:
for each window:
using a disparity value from the disparity map as a disparity of a pixel at a center of the window; and
interpolating disparity values for each other pixel within the window using the disparity at the center; and
warping the transformed frame using the disparity values for each of the pixels for the transformed frame to generate the second view perspective.
16. The non-transitory computer readable storage medium of claim 13 , wherein generating estimated depths comprises applying a spatial-temporal filter to estimated depths for the transformed frame using depth estimates from at least two previous frames within the two dimensional image frame sequence.
17. The non-transitory computer readable storage medium of claim 13 , wherein performing the plurality of actions comprises:
encoding the three dimensional image frame using at least one three dimensional video packing format, including at least one of a field/frame packing, a planar or interleaved mechanism, a top-bottom mechanism, or a side-by-side mechanism.