Scene adaptive temporal filtering
Calculating a temporal filter values to filter video data for noise and ghosting artifacts. A sensor specific noise model may be derived using the video capture settings of the video source to estimate the noise variance for an image, and to determine the amount of the temporal filtering to be applied to a frame or pixel. The global motion and sensor specific noise model may be used to populate filter coefficient look-up tables based on the local motion and luma values of each pixel in the frame. The maximum absolute difference between frames may be used to estimate global motion between two frames such that filter strength may be reduced for frames exhibiting high global motion and increased for frames exhibiting low global motion. According to an embodiment, the increase in temporal filtering between frames may be capped to avoid sudden visible noise differences between frames.
1. A method for processing video data, comprising:
at a temporal noise reduction filter, reducing temporal noise of an input sequence of frames of the video data by:
for each frame,
calculating a global motion as a maximum difference between a current frame and a previous frame;
populating a motion look-up table and a separate luma look-up table using the global motion;
for each pixel of the frame:
calculating a local motion based on the difference between a current pixel and a collocated pixel in a previous frame;
identifying a first filter coefficient from the motion look-up table based on the local motion;
identifying a second filter coefficient from the luma look-up table based on a luma value of the pixel; and
filtering the pixel using both the first and the second filter coefficients.
2. The method of claim 1 , further comprising:
for each frame, calculating a maximum filter strength based on the global motion;
if the maximum filter strength for a frame is larger than the maximum filter strength determined for a previous frame in the sequence by more than a predetermined increase amount, setting the new maximum filter strength to the old maximum filter strength plus the predetermined increase amount.
3. The method of claim 2 , further comprising:
calculating a weight for the motion history.
4. The method of claim 1 , further comprising:
for each frame, receiving a motion history for a previously processed frame.
5. The method of claim 1 , wherein said populating the motion table is further based on the following equation:
k
=
max_filter
_strength
1
+
(
d
(
t
)
β
*
std
)
α
wherein std is the standard deviation of the noise captured by a source of the video data, β is a compensation factor set based on the light level of the frame, α is a control value that determines how fast the filter strength attenuates, d(t) is a potential delta value representing motion between two collocated pixels on adjacent frames, and max_filter_strength is based on the global motion.
6. The method of claim 5 , wherein said populating further comprises reducing a threshold for the max_filter_strength if the frame has high global motion.
7. The method of claim 6 , wherein the frame has high global motion if the global motion is above a predetermined high motion threshold.
8. The method of claim 5 , wherein said populating further comprises increasing a threshold for the max_filter_strength if the frame has low global motion.
9. The method of claim 8 , wherein the frame has low global motion if the global motion is below a predetermined low motion threshold.
10. The method of claim 1 , wherein said populating the motion table is further based on a sensor specific noise model of a source of the video data.
11. The method of claim 10 , wherein said sensor specific noise model is determined using image capture settings of the source.
12. The method of claim 11 , wherein said image capture settings are received by the temporal noise reduction filter as part of the video data.
13. The method of claim 11 , wherein said image capture settings are received by the temporal noise reduction filter on a channel from the source.
14. The method of claim 11 , wherein said image capture settings are received from a controller controlling a plurality of settings for the source.
15. A system comprising:
a temporal noise reduction filter to reduce temporal noise and to receive a sequence of frames wherein:
for each frame, the temporal noise reduction filter:
calculates a global motion as a maximum difference between a current frame and a previous frame, in the sequence of frames and
populates a motion look-up table and a luma look-up table using the global motion;
for each pixel of the frame the temporal noise reduction filter:
calculates a local motion based on a difference between a current pixel and a collocated pixel in a previous frame,
identifies a first filter coefficient from the motion look-up table based on the local motion, and
identifies a second filter coefficient from the luma look-up table based on a luma value of the pixel; and
a filter configured to filter each pixel using both the first and second filter coefficients.
16. The system of claim 15 , wherein the temporal noise reduction filter is further configured to calculate a maximum filter strength for each frame, based on the global motion of the frame and if the maximum filter strength is greater than a predetermined threshold, set the maximum filter strength to the predetermined increase threshold.
17. The system of claim 16 , wherein the predetermined threshold is calculated as the maximum filter strength of a previous frame in the sequence adjusted by a predetermined value.
18. The system of claim 15 , wherein the temporal noise reduction filter further receives a motion history for a previously processed frame.
19. The system of claim 18 , wherein the temporal noise reduction filter is further configured to calculate a weight for the motion history.
20. The system of claim 19 , wherein said weight is adjusted down if the frame has high global motion.
21. The system of claim 19 , wherein said weight is adjusted up if the frame has low global motion.
22. The system of claim 15 , wherein said motion table is populated based on a standard deviation for sensor noise calculated using a sensor specific noise model of a source of the sequence of frames.
23. The system of claim 22 , wherein said sensor specific noise model is determined using image capture settings of the source.
24. The system of claim 23 , wherein said image capture settings are received as part of the video data.
25. The system of claim 23 , further comprising a controller wherein the controller controls a plurality of settings for the source and the temporal noise reduction filter receives image capture settings from the controller.
26. A non-transitory computer readable medium storing program instructions that, when executed by a processing device, cause the device to:
upon receipt of an input sequence of frames of video data, for each frame,
calculate a global motion as a maximum difference between a current frame and a previous frame, in the sequence of frames, and
populate a motion look-up table and a luma look-up table using the global motion, and
for each pixel of the frame:
calculate a local motion based on a difference between a current pixel and a collocated pixel in a previous frame,
identify a first filter coefficient from the motion look-up table based on the local motion, and
identify a second filter coefficient from the luma look-up table based on a luma value of the pixel; and
filter the pixel using both the first and second filter coefficients.
27. The non-transitory computer readable medium of claim 26 , wherein said device is further configured to calculate a maximum filter strength based on the global motion and if the maximum filter strength is greater than a predetermined threshold, setting the new maximum filter strength to the predetermined threshold.
28. The non-transitory computer readable medium of claim 27 , wherein the predetermined threshold is calculated as the maximum filter strength of a previous frame in the sequence adjusted by a predetermined value.
29. The non-transitory computer readable medium of claim 26 , wherein said device is further configured to receive a motion history for a previously processed frame.
30. The non-transitory computer readable medium of claim 29 , wherein said device is further configured to calculate a weight for the motion history.
31. The non-transitory computer readable medium of claim 30 , wherein said weight is lowered if the frame has high global motion.
32. The non-transitory computer readable medium of claim 31 , wherein said weight is increased if the frame has low global motion.
33. The non-transitory computer readable medium of claim 26 , wherein said motion table is populated based on a sensor specific noise model of a source of the video data.
34. The non-transitory computer readable medium of claim 33 , wherein said sensor specific noise model is determined using image capture settings of the source.