Encoder assisted frame rate up conversion using various motion models
An Encoder Assisted Frame Rate Up Conversion (EA-FRUC) system that utilizes various motion models, such as affine models, in addition to video coding and pre-processing operations at the video encoder to exploit the FRUC processing that will occur in the decoder in order to improve the modeling of moving objects, compression efficiency and reconstructed video quality. Furthermore, objects are identified in a way that reduces the amount of information necessary for encoding to render the objects on the decoder device.
1. A method of processing multimedia data comprising:
dividing at least one of first and second video frames into a plurality of partitions;
determining modeling information for at least one object in at least one of the partitions, said modeling information being associated with the first and second video frames;
generating an interpolated frame based on the modeling information; and
generating encoding information based on the interpolated frame, wherein the encoding information and a reference frame are used for generating a video frame temporally co-located with the interpolated frame.
2. The method of claim 1 , wherein determining model information for at least one object in one of the partitions comprises:
determining a block based motion field estimation;
identifying at least one object based on the block based field estimation; and
determining an affine model for the at least one object.
3. The method of claim 1 , further comprising using color features to identify boundaries of the at least one object.
4. The method of claim 1 , further comprising using texture features to identify boundaries of the at least one object.
5. The method of claim 1 , further comprising using pixel domain attributes to identify boundaries of the at least one object.
6. The method of claim 1 , further comprising: determining motion vector erosion information associated with one of the partitions, wherein the transmitted encoding information comprises the motion vector erosion information.
7. The method of claim 1 , wherein the modeling information comprises an affine model.
8. The method of claim 7 , wherein the affine model comprises at least one of translation, rotation, shearing, and scaling motion.
9. The method of claim 1 , wherein the modeling information comprises a global motion model.
10. An apparatus for processing multimedia data comprising: means for dividing at least one of first and second video frames into a plurality of partitions; means for determining modeling information for at least one object in at least one of the plurality of the partitions, said modeling information being associated with the first and second video frames; means for generating an interpolated frame based on the modeling information; and means for generating encoding information based on the interpolated frame, wherein the encoding information and a reference frame are used for generating a video frame temporally co-located with the interpolated frame.
11. The apparatus of claim 10 , wherein said determining means comprises: means for determining block based motion field estimation; means for identifying at least one object based on the block based field estimation; and means for determining an affine model for the at least one object.
12. The apparatus of claim 10 , further comprising using color features to identify boundaries of the at least one object.
13. The apparatus of claim 10 , further comprising using texture features to identify boundaries of the at least one object.
14. The apparatus of claim 10 , further comprising using pixel domain attributes to identify boundaries of the at least one object.
15. The apparatus of claim 10 , further comprising: means for determining motion vector erosion information associated with one of the partitions, wherein the transmitted encoding information comprises the motion vector erosion information.
16. The apparatus of claim 10 , wherein the modeling information comprises an affine model.
17. The apparatus of claim 16 , wherein the affine model comprises at least one of translation, rotation, shearing, and scaling motion.
18. An apparatus for processing multimedia data comprising:
a partitioning module configured to divide at least one of first and second video frames into a plurality of partitions;
a modeling module configured to determine modeling information for at least one object in at least one of the plurality of the partitions, said modeling information being associated with the first and second video frames;
a frame generation module configured to generate an interpolated frame based upon the modeling information;
an encoding module configured to generate encoding information based upon the interpolated frame, wherein the encoding information and a reference frame are used for generating a video frame temporally co-located with the interpolated frame; and
a transmission module configured to transmit the encoding information to a decoder.
19. A non-transitory machine readable medium comprising instructions for processing multimedia data, wherein the instructions upon execution cause a machine to:
divide at least one of first and second video frames into plurality of partitions;
determine modeling information for at least one object in at least one of the plurality of the partitions, said modeling information being associated with the first and second video frames;
generate an interpolated frame based on the modeling information; and
generate encoding information based on the interpolated frame, wherein the encoding information and a reference frame are used for generating a video frame temporally co-located with the interpolated frame.
20. A processor for processing multimedia data, said processor being configured to:
divide at least one of first and second video frames into a plurality of partitions;
determine modeling information for at least one object in at least one of the plurality of the partitions, said modeling information being associated with the first and second video frames;
generate an interpolated frame based on the modeling information; and
generate encoding information based on the interpolated frame, wherein the encoding information and a reference frame are used for generating a video frame temporally co-located with the interpolated frame.