Edge-aware spatio-temporal filtering and optical flow estimation in real time
The disclosure provides an approach for edge-aware spatio-temporal filtering. In one embodiment, a filtering application receives as input a guiding video sequence and video sequence(s) from additional channel(s). The filtering application estimates a sparse optical flow from the guiding video sequence using a novel binary feature descriptor integrated into the Coarse-to-fine PatchMatch method to compute a quasi-dense nearest neighbor field. The filtering application then performs spatial edge-aware filtering of the sparse optical flow (to obtain a dense flow) and the additional channel(s), using an efficient evaluation of the permeability filter with only two scan-line passes per iteration. Further, the filtering application performs temporal filtering of the optical flow using an infinite impulse response filter that only requires one filter state updated based on new guiding video sequence video frames. The resulting optical flow may then be used in temporal edge-aware filtering of the additional channel(s) using the nonlinear infinite impulse response filter.
1. A computer-implemented method for spatio-temporal video filtering, comprising:
receiving an optical flow for a video frame in a video sequence;
temporally filtering the optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame; and
spatially and temporally filtering video frames, from one or more additional video channels, using at least the temporally filtered optical flow.
2. The computer-implemented method of claim 1 , wherein the video frames from the one or more additional video channels are feature maps including one or more of disparity, saliency, or detail values.
3. The computer-implemented method of claim 1 , further comprising estimating the received optical flow by:
estimating a sparse optical flow for the video frame, wherein correspondences between the video frame and a previous frame are determined in the sparse optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame; and
spatially filtering the sparse optical flow using an edge-aware spatial filter.
4. The computer-implemented method of claim 3 , wherein the video frame is divided into a plurality of partially overlapping tiles with overlapping bordering regions, wherein the spatial filtering is separately performed for each of the plurality of partially overlapping tiles, wherein the results of the separately performed spatial filtering are combined.
5. The computer-implemented method of claim 4 , wherein separately performing the spatial filtering for each of the plurality of partially overlapping tiles includes:
determining an approximation of a spatial edge-aware filtering output at a resolution lower than a full resolution;
computing border conditions for the plurality of partially overlapping tiles using at least the approximation of the spatial edge-aware filtering output; and
applying a permeability filter to the plurality of partially overlapping tiles.
6. The computer-implemented method of claim 3 , wherein the binary descriptor is determined with integer arithmetic.
7. The computer-implemented method of claim 3 , wherein the sparse optical flow is spatially filtered using at least one iteration of a permeability filter evaluated per iteration with two scan-line passes and a normalization map.
8. The computer-implemented method of claim 3 , wherein the sparse optical flow is estimated with a modified Coarse-to-fine PatchMatch (CPM) technique using at least the binary descriptor, forward-back checks on coarsest and finest levels, and a quadratic interpolation.
9. The computer-implemented method of claim 1 , wherein the temporal filtering is performed using at least a recursive permeability filter.
10. A non-transitory computer-readable medium storing a program, which, when executed by a processor performs operations for spatio-temporal video filtering, the operations comprising:
receiving an optical flow for a video frame in a video sequence;
temporally filtering the optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame; and
spatially and temporally filtering video frames, from one or more additional video channels, using at least the temporally filtered optical flow.
11. The non-transitory computer-readable medium of claim 10 , wherein the video frames from the one or more additional video channels are feature maps including one or more of disparity, saliency, or detail values.
12. The non-transitory computer-readable medium of claim 10 , the operations further comprising, estimating the received optical flow by:
estimating a sparse optical flow for the video frame, wherein correspondences between the video frame and a previous frame are determined in the sparse optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame; and
spatially filtering the sparse optical flow using an edge-aware spatial filter.
13. The non-transitory computer-readable medium of claim 12 , wherein the video frame is divided into a plurality of partially overlapping tiles with overlapping bordering regions, wherein the spatial filtering is separately performed for each of the plurality of partially overlapping tiles, wherein the results of the separately performed spatial filtering are combined.
14. The non-transitory computer-readable medium of claim 13 , wherein separately performing the spatial filtering for each of the plurality of partially overlapping tiles includes:
determining an approximation of a spatial edge-aware filtering output at a resolution lower than a full resolution;
computing border conditions for the plurality of partially overlapping tiles using at least the approximation of the spatial edge-aware filtering output; and
applying a permeability filter to the plurality of partially overlapping tiles.
15. The non-transitory computer-readable medium of claim 12 , wherein the binary descriptor is determined with integer arithmetic.
16. The non-transitory computer-readable medium of claim 12 , wherein the sparse optical flow is spatially filtered using at least one iteration of a permeability filter evaluated per iteration with two scan-line passes and a normalization map.
17. The non-transitory computer-readable medium of claim 12 , wherein the sparse optical flow is estimated with a modified Coarse-to-fine PatchMatch (CPM) technique using at least the binary descriptor, forward-back checks on coarsest and finest levels, and a quadratic interpolation.
18. A computer-implemented method of determining correspondences between a first image and a second image in a video sequence, comprising:
determining, for each of a plurality of points in the first image and in the second image, binary descriptor values of a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the image; and
comparing the determined binary descriptor values of points in the first image with the determined binary descriptor values of points in the second image to determine correspondences in estimating an optical flow for at least one of the first and second images, wherein images from one or more additional video channels are spatially and temporally filtered using at least the optical flow.
19. The computer-implemented method of claim 18 , wherein the optical flow is spatially filtered using an edge-aware spatial filter and temporally filtered using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the at least one image.
20. A computer-implemented method for spatio-temporal video filtering, comprising:
estimating an optical flow for a video frame in a video sequence, wherein correspondences between the video frame and a previous frame are determined in the optical flow estimation using at least a binary descriptor formulated as a collection of binarized octal orientation maps on spatial bins around a center of an associated region of the video frame;
spatially filtering the estimated optical flow using an edge-aware spatial filter; and
temporally filtering the spatially filtered optical flow using at least an edge-aware filter in time, wherein the edge-aware filter in time uses at least an optical flow from a previous time step to align a filter state from the previous time step to the video frame.