Depth estimation based on interpolation of inverse focus statistics
View Patent ↗Embodiments are directed towards performing depth estimation within a digital camera system based on interpolation of inverse focus statistics. After an image is captured, various statistics or focus measure may be calculated using, for example, a high pass filter. Depth is estimated by interpolating the inverse of the statistics for three positions of focus for the image. The inverse of the statistics, St(n), may be 1/St(n), or 1/St 2 (n), or even 1/St Z (n), where Z≧1. Several approaches to interpolating the inverse values of the statistics to obtain a depth estimate are disclosed, including a general parabolic minimum approach, using a parabolic minimum within a progressive scheme, or within a continuous AF scheme. The depth estimate may then be used for a variety of applications, including automatic focusing, as well as converting 2D images to 3D images.
1. A processor based method, comprising:
receiving a plurality of image frames from an image sensor;
receiving for each of the plurality of image frames, an image statistic and a focusing position;
calculating for each of the plurality of image statistics, an inverse of each of the statistics;
performing a curve fitting determination using the inverse statistics and focusing positions to locate a minimum location on a curve; and
selectively employing the minimum location as an estimated depth.
2. The method of claim 1 , wherein the curve fitting determination employs interpolation of the inverse statistics and focusing positions on a parabolic curve.
3. The method of claim 1 , wherein the plurality of image statistics is obtained by passing each of the plurality of image frames through a high pass filter and further summing filter output values over a predefined window.
4. The method of claim 1 , wherein the curve fitting uses an inverse of a square of the statistics.
5. The method of claim 1 , wherein performing a curve fitting determination further comprises:
finding an index of a global maximum of the statistics as nmax;
setting three points in J according to:
when nmax equals 1, setting J=[1, 2, 3];
else, when nmax equals N, a maximum number statistics received, setting J=[N−2, N−1, N]; and
otherwise, setting J=[nmax−1, nmax, nmax+1];
setting y(i)=1/statistic(J(i)), and setting x(i)=focusing position (J(i)), where 1≦i≦3;
fitting a parabola, y=ax 2 +bx+c, using the set x(i), y(i) points;
when it is determined that the three x(i), y(i) points are not collinear, setting the estimated depth as the minimum location of the parabola, xmin=−b/(2a); and
otherwise, setting the estimated depth as focusing position at nmax.
6. The method of claim 1 , wherein performing a curve fitting determination further comprises:
when the plurality of image frames, n, received are greater than two, n>2, setting a vector J=[n−2, n−1, n];
when (the statistics for frame n−1 is greater than the statistics for frame n−2) and (the statistics for frame n−1 is greater than the statistics for frame n):
setting y(i)=1/statistics (J(i)), and x(i)=focusing positions (J(i)), for 1≦i≦3,
performing the curve fitting using a parabola, y=ax2+bx+c and the three x(i), y(i) points, to employ the minimum location, xmin=−b/(2a), as the estimated depth;
else, when (n=3 and the statistics for frame 1 is greater than a maximum of (the statistics of frame 2, statistics of frame 3)):
setting the estimated depth as a focusing position of frame 1; and
else, when (n=N, a maximum number of image frames, and the statistics for frame N is greater than a maximum of (the statistics of frame N−1, statistics of frame N−2)):
setting the estimated depth as the focusing position of frame N.
7. The method of claim 1 , wherein performing a curve fitting determination and wherein receiving for each of a plurality of image frames, a image statistic and a focusing position, further comprises:
receiving a first image statistic for a first image frame at a first focusing position, x(1);
relocating a focusing position for the image sensor by d steps ahead of the first focusing position, receiving a second image frame having a second image statistic and wherein a second focusing position is determined as x(2)=x(1)+d;
when the first image statistic is greater than the second image statistic, setting a third focusing position, as x(3)=x(1)−d;
else, when the first image statistic is less than the second image statistic, setting the third focusing position, as x(3)=x(2)+d;
otherwise, setting the third focusing position, as x(3)=x(1)+(d/2);
receiving a third image frame and related statistic using the third focusing position;
setting y(i)=1/statistics(i), for 1≦i≦3;
fitting a parabola, y=ax 2 +bx+c, using the set x(i), y(i) points;
finding the minimum location, of xmin=−b/(2a);
when minimum of (x(1), x(2), x(3))≦xmin≦maximum of (x(1), x(2), x(3)), using xmin as the estimated depth; else
identifying two points in [x(1), x(2), x(3)] which are closest to xmin, and using the identified two points to set a new x(1) and x(2), setting a new focusing position x(3)=xmin, acquiring a new third image frame and related statistics using the new focusing position x(3) and further setting a new y(i)=1/statistics(i), for 1≦i≦3.
8. An image system, comprising:
an image sensor device that is configured to receive image frames; and
one or more circuits having a plurality of components thereon operating to perform a plurality of actions, including:
receiving for each of a plurality of image frames, an image statistic and a focusing position;
calculating for each of the plurality of image statistics, an inverse of each of the statistics;
performing a curve fitting determination using the inverse statistics and focusing positions to locate a minimum location on a curve; and
selectively employing the minimum location as an estimated depth.
9. The image system of claim 8 , wherein the curve fitting determination employs interpolation of the inverse statistics and focusing positions on a parabolic curve.
10. The image system of claim 8 , wherein the plurality of image statistics is obtained by passing each of the plurality of image frames through a high pass filter and further summing filter output values over a predefined window.
11. The image system of claim 8 , wherein the curve fitting uses an inverse of a square of the statistics.
12. The image system of claim 8 , wherein performing a curve fitting determination further comprises:
finding an index of a global maximum of the statistics as nmax;
setting three points in J according to:
when nmax equals 1, setting J=[1, 2, 3];
else, when nmax equals N, a maximum number statistics received, setting J=[N−2, N−1, N]; and
otherwise, setting J=[nmax−1, nmax, nmax+1];
setting y(i)=1/statistic(J(i)), and setting x(i)=focusing position (J(i)), where 1≦i≦3;
fitting a parabola, y=ax 2 +bx+c, using the set x(i), y(i) points;
when it is determined that the three x(i), y(i) points are not collinear, setting the estimated depth as the minimum location of the parabola, xmin=−b/(2a); and
otherwise, setting the estimated depth as focusing position at nmax.
13. The image system of claim 8 , wherein performing a curve fitting determination further comprises:
when the plurality of image frames, n, received are greater than two, n>2, setting a vector J=[n−2, n−1, n];
when (the statistics for frame n−1 is greater than the statistics for frame n−2) and (the statistics for frame n−1 is greater than the statistics for frame n):
setting y(i)=1/statistics (J(i)), and x(i)=focusing positions (J(i)), for 1≦i≦3,
performing the curve fitting using a parabola, y=ax2+bx+c and the three x(i), y(i) points, to employ the minimum location, xmin=−b/(2a), as the estimated depth;
else, when (n=3 and the statistics for frame 1 is greater than a maximum of (the statistics of frame 2, statistics of frame 3)):
setting the estimated depth as a focusing position of frame 1; and
else, when (n=N, a maximum number of image frames, and the statistics for frame N is greater than a maximum of (the statistics of frame N−1, statistics of frame N−2)):
setting the estimated depth as the focusing position of frame N.
14. The image system of claim 8 , wherein performing a curve fitting determination and wherein receiving for each of a plurality of image frames, a image statistic and a focusing position, further comprises:
receiving a first image statistic for a first image frame at a first focusing position, x(1);
relocating a focusing position for the image sensor device by d steps ahead of the first focusing position, receiving a second image frame having a second image statistic and wherein a second focusing position is determined as x(2)=x(1)+d;
when the first image statistic is greater than the second image statistic, setting a third focusing position, as x(3)=x(1)−d;
else, when the first image statistic is less than the second image statistic, setting the third focusing position, as x(3)=x(2)+d;
otherwise, setting the third focusing position, as x(3)=x(1)+(d/2);
receiving a third image frame and related statistic using the third focusing position;
setting y(i)=1/statistics(i), for 1≦i≦3;
fitting a parabola, y=ax 2 +bx+c, using the set x(i), y(i) points;
finding the minimum location, of xmin=−b/(2a);
when minimum of (x(1), x(2), x(3))≦xmin≦maximum of (x(1), x(2), x(3)), using xmin as the estimated depth; else
identifying two points in [x(1), x(2), x(3)] which are closest to xmin, and using the identified two points to set a new x(1) and x(2), setting a new focusing position x(3)=xmin, acquiring a new third image frame and related statistics using the new focusing position x(3) and further setting a new y(i)=1/statistics(i), for 1≦i≦3.
15. A non-transitory storage device having stored thereon a plurality of computer-executable instructions that when installed on a digital camera performs actions, comprising:
receiving a plurality of image frames from an image sensor;
receiving for each of the plurality of image frames, a image statistic and a focusing position;
calculating for each of the plurality of image statistics, an inverse of each of the statistics;
performing a curve fitting determination using the inverse statistics and focusing positions to locate a minimum location on a curve; and
selectively employing the minimum location as an estimated depth.
16. The storage device of claim 15 , wherein the curve fitting determination employs interpolation of the inverse statistics and focusing positions on a parabolic curve.
17. The storage device of claim 15 , wherein the plurality of image statistics is obtained by passing each of the plurality of image frames through a high pass filter.
18. The storage device of claim 15 , wherein the curve fitting uses an inverse of a square of the statistics.
19. The storage device of claim 15 , wherein the statistics are obtained using a difference operator.
20. The storage device of claim 15 , wherein performing a curve fitting determination further comprises employ a parabolic minimum technique with a progressive image capture to obtain the estimated depth.