Content-adaptive background skipping for region-of-interest video coding
The disclosure is directed to techniques for content-adaptive background skipping for region-of-interest (ROI) video coding. The techniques may be useful in video telephony (VT) applications such as video streaming and videoconferencing, and especially useful in low bit-rate wireless communication applications, such as mobile VT. The disclosed techniques analyze content information of a video frame to dynamically determine whether to skip a non-ROI area within the frame. For example, the skipping determination may be based on content activity, such as ROI shape deformation, ROI motion, non-ROI motion, non-ROI texture complexity, and accumulated distortion due to non-ROI skipping. The skip determination may operate in conjunction with either frame-level or macroblock-level bit allocation.
1. A method for encoding video at an encoding module, comprising:
obtaining a video frame;
encoding a region of interest (ROI) and a non-ROI area within the video frame by allocating bits between the ROI and the non-ROI area;
determining content activity in the ROI;
determining content activity in the non-ROI area; and
determining whether to skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI exceeds a first threshold,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.
2. The method of claim 1 , wherein the content activity of the ROI includes shape deformation and motion in the ROI.
3. The method of claim 1 , wherein the content activity of the non-ROI area includes motion and texture complexity in the non-ROI area.
4. The method of claim 1 , further comprising allocating encoding bits between the ROI and the non-ROI area when encoding of the non-ROI area is not skipped.
5. The method of claim 4 , wherein allocating encoding bits includes allocating the encoding bits based on a frame-level bit budget, the method further comprising adjusting the frame-level bit budget based on historical and predictive skipping statistics for non-ROI areas of the video frame and other video frames.
6. The method of claim 5 , further comprising applying a Bayesian model to the historical skipping statistics to obtain the predictive skipping statistics.
7. The method of claim 4 , wherein allocating encoding bits includes allocating encoding bits in a ρ-domain at a macroblock level within the video frame.
8. The method of claim 4 , wherein allocating encoding bits includes allocating the encoding bits based on a ρ-domain bit budget.
9. The method of claim 8 , wherein allocating encoding bits includes applying a bit allocation scheme based on a rate control strategy that determines the ρ-domain bit budget based on a texture complexity of the ROI and non-ROI areas within the frame.
10. The method of claim 8 , wherein allocating encoding bits includes applying a bit allocation scheme based on a rate control strategy that estimates future skipping events based on statistics and patterns of non-ROI skipping history over previous frames within a video sequence, and then determines the ρ-domain bit budget based on the estimation.
11. A device comprising:
a region-of-interest mapper that generates a definition of a region of interest (ROI) within a video frame;
a video encoder that encodes the ROI and a non-ROI area within the video frame by allocating bits between the ROI and the non-ROI area; and
a skipping module that determines:
content activity in the ROI;
content activity in the non-ROI area; and
whether the video encoder will skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI exceeds a first threshold,
wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein the skipping module directs the video encoder to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.
12. The device of claim 11 , wherein the content activity of the ROI includes shape deformation and motion in the ROI.
13. The device of claim 11 , wherein the content activity of the non-ROI area includes motion and texture complexity in the non-ROI area.
14. The device of claim 11 , further comprising a bit allocation module that allocates encoding bits between the ROI and the non-ROI area when encoding of the non-ROI area is not skipped.
15. The device of claim 14 , wherein the bit allocation module allocates the encoding bits based on a frame-level bit budget, the device further comprising a budget adjustment module that adjusts the frame-level bit budget based on historical and predictive skipping statistics for non-ROI areas of the video frame and other video frames.
16. The device of claim 15 , wherein the budget adjustment module applies a Bayesian model to the historical skipping statistics to obtain the predictive skipping statistics.
17. The device of claim 14 , wherein the bit allocation module allocates encoding bits in a ρ-domain at the macroblock level within the video frame.
18. The device of claim 14 , wherein the bit allocation module allocates the encoding bits based on a ρ-domain bit budget.
19. The device of claim 18 , wherein the bit allocation module applies a bit allocation scheme based on a rate control strategy that determines the ρ-domain budget based on a texture complexity of the ROI and non-ROI areas within the frame.
20. The device of claim 18 , wherein the bit allocation module applies a bit allocation scheme based on a rate control strategy that estimates future skipping events based on statistics and patterns of non-ROI skipping history over previous frames within a video sequence, and then determines the ρ-domain bit budget based on the estimation.
21. A non-transitory computer-readable medium encoded with a computer program comprising instructions to cause a processor to:
obtain a video frame;
encode a region of interest (ROI) and a non-ROI area within the video frame by allocating bits between the ROI and non-ROI area;
determine content activity in the ROI;
determine content activity in the non-ROI area; and
determine whether to skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI exceeds a first threshold,
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein encoding of the non-ROI area is skipped when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.
22. The non-transitory computer-readable medium of claim 21 , wherein the content activity of the ROI includes shape deformation and motion in the ROI.
23. The non-transitory computer-readable medium of claim 21 , wherein the content activity of the non-ROI area includes motion and texture complexity in the non-ROI area.
24. The non-transitory computer-readable medium of claim 21 , wherein the instructions cause the processor to allocate encoding bits between the ROI and the non-ROI area when encoding of the non-ROI area is not skipped.
25. The non-transitory computer-readable medium of claim 24 , wherein the instructions cause the processor to allocate encoding bits based on a frame-level bit budget, and adjust the frame-level bit budget based on historical and predictive skipping statistics for non-ROI areas of the video frame and other video frames.
26. The non-transitory computer-readable medium of claim 25 , wherein the instructions cause the processor to apply a Bayesian model to the historical skipping statistics to obtain the predictive skipping statistics.
27. The non-transitory computer-readable medium of claim 24 , wherein the instructions cause the processor to allocate encoding bits in a ρ-domain at a macroblock level within the video frame.
28. The non-transitory computer-readable medium of claim 24 , wherein the instructions cause the processor to allocate the encoding bits based on a ρ-domain bit budget.
29. The non-transitory computer-readable medium of claim 28 , wherein the instructions cause the processor to apply a bit allocation scheme based on a rate control strategy that determines the ρ-domain bit budget based on a texture complexity of the ROI and non-ROI areas within the frame.
30. The non-transitory computer-readable medium of claim 28 , wherein the instructions cause the processor to apply a bit allocation scheme based on a rate control strategy that estimates future skipping events based on statistics and patterns of non-ROI skipping history over previous frames within a video sequence, and then determines the p-domain bit budget based on the estimation.
31. A device comprising:
a region-of-interest mapping means that generates a definition of a region of interest (ROI) within a video frame;
a video encoding means that encodes the ROI and a non-ROI area within the video frame by allocating bits between the ROI and the non-ROI area; and
a skipping means that determines:
content activity in the ROI;
content activity in the non-ROI area; and
whether the video encoding means will skip encoding of the non-ROI area of the video frame based on a content activity of the ROI, a content activity of the non-ROI area, and an accumulated distortion due to skipping of encoding of non-ROI areas in one or more other video frames,
wherein the skipping means directs the video encoding means to skip encoding of the non-ROI area when the content activity of the ROI exceeds a first threshold,
wherein the skipping means directs the video encoding means to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold and the content activity of the non-ROI area does not exceed a second threshold, and
wherein the skipping means directs the video encoding means to skip encoding of the non-ROI area when the content activity of the ROI does not exceed the first threshold, the content activity within the non-ROI area exceeds a second threshold, and the accumulated distortion due to skipping of encoding of the other video frames does not exceed a third threshold.