Encoder-assisted adaptive video frame interpolation
The disclosure is directed to techniques for encoder-assisted adaptive interpolation of video frames. According to the disclosed techniques, an encoder generates information to assist a decoder in interpolation of a skipped video frame, i.e., an S frame. The information permits the decoder to reduce visual artifacts in the interpolated frame and thereby achieve improved visual quality. The information may include interpolation equation labels that identify selected interpolation equations to be used by the decoder for individual video blocks. As an option, to conserve bandwidth, the equation labels may be transmitted for only selected video blocks that meet a criterion for encoder-assisted interpolation. Other video blocks without equation labels may be interpolated according to a default interpolation technique.
1. A video decoding method comprising:
receiving information encoding a video frame;
deciding whether said video frame is an intentionally skipped video frame based on one or more dynamic frame skipping criteria, said information comprising motion vectors encoding blocks of the video frame and further information specifying one or more interpolation techniques to assist in interpolation of one or more video blocks within said video frame if said video frame is an intentionally skipped video frame; and
applying the information to interpolate said one or more video blocks of the intentionally skipped video frame using said one or more interpolation techniques, wherein said one or more interpolation techniques include one or more motion calculations using one or more forward motion vectors and one or more backward motion vectors, wherein said one or more forward motion vectors are included in the information for the intentionally skipped video frame, and wherein said one or more backward motion vectors are not included in the information for the intentionally skipped video frame but are included in information for a predictive video frame received subsequent to the intentionally skipped video frame.
2. The method of claim 1 , wherein the information specifies a single interpolation equation to be used in interpolation of all of the video blocks within the intentionally skipped video frame, the method further comprising applying the specified interpolation equation to interpolate all of the video blocks within the intentionally skipped video frame.
3. The method of claim 1 , wherein the information specifies different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the method further comprising applying the specified different interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
4. The method of claim 3 , wherein the selected video blocks comprise substantially less than all of the video blocks within the intentionally skipped video frame, thereby reducing an amount of the information.
5. The method of claim 1 , wherein the information includes an equation label map specifying different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the method further comprising applying the specified interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
6. The method of claim 1 , wherein the information specifies interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, and wherein differences between the selected video blocks within the intentionally skipped video frame and corresponding video blocks within another video frame exceed a threshold value.
7. The method of claim 1 , wherein the information includes motion vectors representing motion between the intentionally skipped video frame, a previous frame, and a subsequent frame, the method further comprising applying the motion vectors to interpolate the intentionally skipped video frame.
8. The method of claim 1 , wherein the information includes difference flags indicating differences between the intentionally skipped video frame, a previous frame, and a subsequent frame, the method further comprising applying the difference flags to interpolate the intentionally skipped video frame.
9. The method of claim 8 , wherein the difference flags include sum of absolute difference (SAD) flags and directional flags indicating relative SAD values among neighboring blocks.
10. A video decoder comprising an interpolation unit that receives information encoding a video frame, decides whether said video frame is an intentionally skipped video frame based on one or more dynamic frame skipping criteria, said information comprising motion vectors encoding blocks of the video frame and further information specifying one or more interpolation techniques to assist in interpolation of one or more video blocks within said video frame if said video frame is an intentionally skipped video frame, and applies the information to interpolate said one or more video blocks of the intentionally skipped video frame using said one or more interpolation techniques, wherein said one or more interpolation techniques include one or more motion calculations using one or more forward motion vectors and one or more backward motion vectors, wherein said one or more forward motion vectors are included in the information for the intentionally skipped video frame, and wherein said one or more backward motion vectors are not included in the information for the intentionally skipped video frame but are included in information for a predictive video frame received subsequent to the intentionally skipped video frame.
11. The video decoder of claim 10 , wherein the information specifies a single interpolation equation to be used in interpolation of all of the blocks within the intentionally skipped video frame.
12. The video decoder of claim 10 , wherein the information specifies different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame.
13. The video decoder of claim 12 , wherein the selected video blocks comprise substantially less than all of the video blocks within the intentionally skipped video frame, thereby reducing an amount of the information.
14. The video decoder of claim 10 , wherein the information includes an equation label map specifying different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame.
15. The video decoder of claim 10 , wherein the information specifies interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, and wherein differences between the selected video blocks within the intentionally skipped video frame and corresponding video blocks within another video frame exceed a threshold value.
16. The video decoder of claim 10 , wherein the information includes motion vectors representing motion between the intentionally skipped video frame, a previous frame, and a subsequent frame.
17. The video decoder of claim 10 , wherein the information includes difference flags indicating differences between the intentionally skipped video frame, a previous frame, and a subsequent frame.
18. The video decoder of claim 17 , wherein the difference flags include sum of absolute difference (SAD) flags and directional flags indicating relative SAD values among neighboring blocks.
19. A video decoder comprising:
means for receiving information encoding a video frame;
means for deciding whether said video frame is an intentionally skipped video frame based on one or more dynamic frame skipping criteria, said information comprising motion vectors encoding blocks of the video frame and further information specifying one or more interpolation techniques to assist in interpolation of one or more video blocks within said video frame if said video frame is an intentionally skipped video frame; and
means for applying the information to interpolate said one or more video blocks of the intentionally skipped video frame using said one or more interpolation techniques, wherein said one or more interpolation techniques include one or more motion calculations using one or more forward motion vectors and one or more backward motion vectors, wherein said one or more forward motion vectors are included in the information for the intentionally skipped video frame, and wherein said one or more backward motion vectors are not included in the information for the intentionally skipped video frame but are included in information for a predictive video frame received subsequent to the intentionally skipped video frame.
20. The video decoder of claim 19 , wherein the information specifies a single interpolation equation to be used in interpolation of all of the video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified interpolation equation to interpolate all of the video blocks within the intentionally skipped video frame.
21. The video decoder of claim 19 , wherein the information specifies different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified different interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
22. The video decoder of claim 21 , wherein the selected video blocks comprise substantially less than all of the video blocks within the intentionally skipped video frame, thereby reducing an amount of the information.
23. The video decoder of claim 19 , wherein the information includes an equation label map specifying different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
24. The video decoder of claim 19 , wherein the information specifies interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, and wherein differences between the selected video blocks within the intentionally skipped video frame and corresponding video blocks within another video frame exceed a threshold value.
25. The video decoder of claim 19 , wherein the information includes motion vectors representing motion between the intentionally skipped video frame, a previous frame, and a subsequent frame, the video decoder further comprising means for applying the motion vectors to interpolate the intentionally skipped video frame.
26. The video decoder of claim 19 , wherein the information includes difference flags indicating differences between the intentionally skipped video frame, a previous frame, and a subsequent frame, the video decoder further comprising means for applying the difference flags to interpolate the intentionally skipped video frame.
27. The video decoder of claim 26 , wherein the difference flags include sum of absolute difference (SAD) flags and directional flags indicating relative SAD values among neighboring blocks.
28. A computer-readable medium having stored thereon instructions to cause a processor to:
receive information encoding a video frame;
decide whether said video frame is an intentionally skipped video frame based on one or more dynamic frame skipping criteria, said information comprising motion vectors encoding blocks of the video frame and further information specifying one or more interpolation techniques to assist in interpolation of one or more video blocks within said video frame if said video frame is an intentionally skipped video frame; and
apply the information to interpolate said one or more video blocks of the intentionally skipped video frame using said one or more interpolation techniques, wherein said one or more interpolation techniques include one or more motion calculations using one or more forward motion vectors and one or more backward motion vectors, wherein said one or more forward motion vectors are included in the information for the intentionally skipped video frame, and wherein said one or more backward motion vectors are not included in the information for the intentionally skipped video frame but are included in information for a predictive video frame received subsequent to the intentionally skipped video frame.
29. The computer readable medium of claim 28 , wherein the information specifies a single interpolation equation to be used in interpolation of all of the video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified interpolation equation to interpolate all of the video blocks within the intentionally skipped video frame.
30. The computer readable medium of claim 28 , wherein the information specifies different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified different interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
31. The computer readable medium of claim 30 , wherein the selected video blocks comprise substantially less than all of the video blocks within the intentionally skipped video frame, thereby reducing an amount of the information.
32. The computer readable medium of claim 28 , wherein the information includes an equation label map specifying different interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, the video decoder further comprising means for applying the specified interpolation equations to interpolate the selected video blocks within the intentionally skipped video frame.
33. The computer readable medium of claim 28 , wherein the information specifies interpolation equations to be used in interpolation of selected video blocks within the intentionally skipped video frame, and wherein differences between the selected video blocks within the intentionally skipped video frame and corresponding video blocks within another video frame exceed a threshold value.
34. The computer readable medium of claim 28 , wherein the information includes motion vectors representing motion between the intentionally skipped video frame, a previous frame, and a subsequent frame, the video decoder further comprising means for applying the motion vectors to interpolate the intentionally skipped video frame.
35. The computer readable medium of claim 28 , wherein the information includes difference flags indicating differences between the intentionally skipped video frame, a previous frame, and a subsequent frame, the video decoder further comprising means for applying the difference flags to interpolate the intentionally skipped video frame.
36. The computer readable medium of claim 35 , wherein the difference flags include sum of absolute difference (SAD) flags and directional flags indicating relative SAD values among neighboring blocks.