Digital processing method and system for determination of optical flow
A method and system for determining an optical flow field between a pair of images is disclosed. Each of the pair of images is decomposed into image pyramids using a non-octave pyramid factor. The pair of decomposed images is transformed at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant. Discrete-time derivatives of the second derivative image representations are estimated. An optical flow estimation process is applied to the discrete-time derivatives to produce a raw optical flow field. The raw optical flow field is scaled by the non-octave pyramid factor. The above-cited steps are repeated for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations are warped by a previous raw optical flow estimation.
1. A method for determining an optical flow field between a pair of images, comprising the steps of:
(a) decomposing each of the pair of images into image pyramids using a non-octave pyramid factor;
(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;
(c) estimating discrete-time derivatives of the second derivative image representations;
(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;
(e) scaling the raw optical flow field by the non-octave pyramid factor, and
(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations in step (c) are warped by a previous raw optical flow estimation.
2. The method of claim 1 , further comprising the step of (g) transforming the second derivative image representations from color-space to a scalar color-distance space.
3. The method of claim 2 , wherein step (g) is implemented using the Laplacian of the vector subtraction of color-distances.
4. The method of claim 3 , wherein the Laplacian operator is applied to intensities of all color channels of the second-derivative image representations separately.
5. The method of claim 1 , wherein the non-octave pyramid factor is chosen to be between 1.5 and 2.0.
6. The method of claim 5 , further comprising employing a pre-scaling step via bilinear interpolation to ensure that the highest resolution pyramid scale devolves to integer image sizes of (W′,H′), wherein W′ and H′ are given by
W ′=floor( W/N )×pyramid_factor N
H ′=floor( WM )×pyramid_factor N .
7. The method of claim 1 , wherein the first pyramid scale is the lowest resolution pyramid scale and another pyramid scale is next higher resolution pyramid scale.
8. The method of claim 1 , wherein step (c) is accomplished using a spatiotemporal gradient estimation method.
9. The method of claim 1 , wherein the optical flow estimation process employs a least-squares-fit method.
10. The method of claim 1 , further comprising the step of processing the raw optical flow field with at least one non-linear filter that imposes a non-linear, piecewise-linearity constraint on the optical flow field.
11. The method of claim 10 , wherein the at least one non-linear filter is at least one of a total variation minimization (L1 and L2 norm) filter, an anisotropic diffusion filter, an elliptic steerable filter that reacts to local image structures, and a kernel regression filter.
12. The method of claim 1 , further comprising the steps of performing multiple-pixel motion-compensation and a sum-of-square-differences calculation on the raw optical flow field to check inverse consistency of the raw optical flow field on a per-pixel-sample basis.
13. The method of claim 1 , further comprising the step of applying a 2D structure tensor eigensystem analysis to the discrete-time derivatives to provide a feedback mechanism for selecting a non-octave (2.0) pyramid scaling factor between 1.5 and 2.0 at the subsequent lower resolution pyramid scales.
14. The method of claim 13 , wherein when small eigenvalues below a threshold percentage of pixels for a single pyramid scale in excess of other pyramid scales is encountered during the 2D structure tensor eigensystem analysis, a finer pyramid scale is used for at least one of the subsequent pyramid scales.
15. The method of claim 1 , wherein the raw optical field flow is processed in reverse order.
16. The method of claim 15 , wherein a motion vector is used to sample the second image of the pair of images in five or more locations of a Robert's Cross configuration.
17. The method of claim 1 , further comprising the steps of processing the raw optical flow field for all pyramid scales by a (non-linear) 3×3 median filter operation to eliminate single-pixel outliers and processing the filtered raw optical flow field with an L2 norm total variation operator for a variable number of iterations.
18. A non-transitory computer-readable medium storing computer code determining as optical flow field between a pair of images, the code being executed by at least one processor, wherein the computer code comprises code for:
(a) decomposing each of the pair of images into image pyramids using a non-octave pyramid factor;
(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;
(c) estimating discrete-time derivatives of the second derivative image representations;
(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;
(e) scaling the raw optical flow field by the non-octave pyramid factor; and
(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations in step (c) are warped by a previous raw optical flow estimation.
19. The non-transitory computer-readable medium of claim 18 , further comprising code for (g) transforming the second derivative image representations from color-space to a scalar color-distance space.
20. The non-transitory computer-readable medium of claim 19 , wherein (g) is implemented using the Laplacian of the vector subtraction of color-distances.
21. The non-transitory computer-readable medium of claim 18 , wherein the non-octave pyramid factor is chosen to be between 1.5 and 2.0.
22. The non-transitory computer-readable medium of claim 18 , wherein the first pyramid scale is the lowest resolution pyramid scale and another pyramid scale is next higher resolution pyramid scale.
23. The non-transitory computer-readable medium of claim 18 , wherein (c) is accomplished using a spatiotemporal gradient estimation method.
24. The non-transitory computer-readable medium of claim 18 , wherein the optical flow estimation process employs a least-squares-fit method.
25. The non-transitory computer-readable medium of claim 18 , further comprising code for processing the raw optical flow field with at least one non-linear filter that imposes a non-linear, piecewise-linearity constraint on the optical flow field.
26. The non-transitory computer-readable medium of claim 25 , wherein the at least one non-linear filter is at least one of a total variation minimization (L1 and L2 norm) filter, an anisotropic diffusion filter, an elliptic steerable filter that reacts to local image structures, and a kernel regression filter.
27. The non-transitory computer-readable medium of claim 18 , further comprising code for applying a 2D structure tensor eigensystem analysis to the discrete-time derivatives to provide a feedback mechanism for selecting a non-octave (2.0) pyramid scaling factor between 1.5 and 2.0 at the subsequent lower resolution pyramid scales.
28. The non-transitory computer-readable medium of claim 18 , wherein the raw optical field flow is processed in reverse order.
29. The non-transitory computer-readable medium of claim 28 , wherein a motion vector is used to sample the second image of the pair of images in five or more locations of a Robert's Cross configuration.
30. A method for determining an optical flow field between a pair of images, comprising the steps of:
(a) decomposing each of the pair of images into image pyramids using a non-octave pyramid factor, wherein the non-octave pyramid factor is chosen to be between 1.5 and 2.0;
(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;
(c) estimating discrete-time derivatives of the second derivative image representations;
(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;
(e) scaling the raw optical flow field by the non-octave pyramid factor;
(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations in step (c) are warped by a previous raw optical flow estimation; and
employing a pre-scaling step via bilinear interpolation to ensure that the highest resolution pyramid scale devolves to integer image sizes of (W′,H′), wherein W′ and H′ are given by
W ′=floor( W/N )×pyramid_factor N
H ′=floor( WM )×pyramid_factor N
wherein N is the total number of pyramid scales.
31. A method for determining an optical flow field between a pair of images, comprising the steps of:
(a) decomposing each of the pair of images into image pyramids using a non-octave pyramid factor;
(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;
(c) estimating discrete-time derivatives of the second derivative image representations;
(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;
(e) scaling the raw optical flow field by the non-octave pyramid factor;
(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations in step (c) are warped by a previous raw optical flow estimation; and
applying a 2D structure tensor eigensystem analysis to the discrete-time derivatives to provide a feedback mechanism for selecting a non-octave (2.0) pyramid scaling factor between 1.5 and 2.0 at the subsequent lower resolution pyramid scales,
wherein when small eigenvalues below a threshold percentage of pixels for a single pyramid scale in excess of other pyramid scales is encountered during the 2D structure tensor eigensystem analysis, a finer pyramid scale is used for at least one of the subsequent pyramid scales.