System and method for frame rate up-conversion of video data

1. A computer-implemented method for performing frame rate up-conversion of video data including a sequence of image frames, comprising:

determining, by a video processor, a set of motion vectors of a target frame relative to a plurality of reference frames, wherein the target frame is to be generated and interpolated into the sequence of image frames;

performing, by the video processor, a motion vector classification on the set of motion vectors to generate a target object map for the target frame;

projecting, by the video processor, the target object map onto the plurality of reference frames to generate a plurality of reference object maps based on the set of motion vectors; and

detecting, by the video processor, an occlusion area in the target frame based on the set of motion vectors, the target object map, and the plurality of reference object maps.

2. The method of claim 1 , wherein performing the motion vector classification on the set of motion vectors to generate the target object map for the target frame comprises:

performing the motion vector classification on the set of motion vectors to detect one or more objects in the target frame;

generating the target object map to include the one or more objects; and

determining one or more relative depth values of the one or more objects in the target object map.

3. The method of claim 2 , wherein performing the motion vector classification on the set of motion vectors to detect the one or more objects in the target frame comprises:

classifying the set of motion vectors into one or more groups of motion vectors; and

for each group of motion vectors, determining an object corresponding to the group of motion vectors.

4. The method of claim 3 , wherein determining the object corresponding to the group of motion vectors comprises:

determining one or more target blocks of the target frame, each of the one or more target blocks having a respective motion vector being classified into the group of motion vectors; and

determining the object to be an image area comprising the one or more target blocks of the target frame.

5. The method of claim 1 , wherein the target frame comprises a plurality of target blocks, and projecting the target object map onto the plurality of reference frames to generate the plurality of reference object maps comprises:

for each reference frame,

projecting the plurality of target blocks onto the reference frame to generate a plurality of block projections based on motion vectors of the plurality of target blocks relative to the reference frame, respectively; and

combining the plurality of block projections to generate a reference object map for the reference frame based on the target object map.

6. The method of claim 5 , wherein each of the plurality of target blocks has a variable block size.

7. The method of claim 5 , wherein the reference frame is divided into a plurality of reference blocks, and combining the plurality of block projections to generate the reference object map for the reference frame comprises:

for a reference block where two or more block projections of two or more target blocks overlap,

determining a set of relative depth values associated with the two or more target blocks;

determining a minimal relative depth value among the set of relative depth values;

identifying, from the two or more block projections, a block projection associated with a target block having the minimal relative depth value;

determining that the reference block is covered by an object associated with the target block having the minimal relative depth value such that the object is identified in the reference object map; and

determining a relative depth value of the reference block to be that of the object.

8. The method of claim 1 , wherein detecting the occlusion area in the target frame comprises detecting a set of occluded target blocks from a plurality of target blocks in the target frame.

9. The method of claim 8 , wherein the occlusion area comprises a covered occlusion area that includes one or more occluded target blocks having a covered occlusion status, an uncovered occlusion area that includes one or more occluded target blocks having an uncovered occlusion status, or a combined occlusion area that includes one or more occluded target blocks having a combined occlusion status.

10. The method of claim 8 , wherein

the plurality of reference frames comprise a first previous frame preceding the target frame and a first next frame subsequent to the target frame;

the plurality of reference object maps comprise a first previous object map for the first previous frame and a first next object map for the first next frame; and

detecting the set of occluded target blocks comprises:

for each target block in the target frame, determining a first occlusion detection result for the target block, wherein the first occlusion detection result indicates whether the target block is an occluded target block relative to the first previous and next frames.

11. The method of claim 10 , wherein determining the first occlusion detection result for the target block comprises:

determining, based on a motion vector of the target block relative to the first previous frame, a first previous block of the first previous frame that corresponds to the target block;

determining a relative depth value of the first previous block based on the first previous object map;

determining, based on a motion vector of the target block relative to the first next frame, a first next block of the first next frame that corresponds to the target block;

determining a relative depth value of the first next block based on the first next object map; and

determining the first occlusion detection result for the target block based on a relative depth value of the target block, the relative depth value of the first previous block, and the relative depth value of the first next block.

12. The method of claim 11 , wherein determining the first occlusion detection result for the target block comprises:

responsive to the relative depth value of the target block being not greater than the relative depth value of the first previous block and being greater than the relative depth value of the first next block, determining that the target block is an occluded target block having a covered occlusion status relative to the first previous and next frames, a matched block of the target block being the first previous block in the first previous frame.

13. The method of claim 11 , wherein determining the first occlusion detection result for the target block comprises:

responsive to the relative depth value of the target block being greater than the relative depth value of the first previous block and being not greater than the relative depth value of the first next block, determining that the target block is an occluded target block having an uncovered occlusion status relative to the first previous and next frames, a matched block of the target block being the first next block in the next previous frame.

14. The method of claim 11 , wherein determining the first occlusion detection result for the target block comprises:

responsive to the relative depth value of the target block being greater than the relative depth value of the first previous block and also greater than the relative depth value of the first next block, determining that the target block is an occluded target block having a combined occlusion status relative to the first previous and next frames, the target block having no matched block in the first previous frame and the first next frame.

15. The method of claim 14 , wherein

the plurality of reference frames further comprise a second previous frame preceding the first previous frame and a second next frame subsequent to the first next frame;

the plurality of reference object maps further comprise a second previous object map for the second previous frame and a second next object map for the second next frame; and

the method further comprises:

determining a second occlusion detection result for the target block, wherein the second occlusion detection result indicates whether the target block is an occluded target block relative to the second previous and next frames.

16. The method of claim 1 , wherein

the plurality of reference frames comprise a first previous frame preceding the target frame and a first next frame subsequent to the target frame; and

determining the set of motion vectors of the target block relative to the plurality of reference frames, respectively, comprises:

dividing the target frame into a plurality of target blocks, and

for each target block, determining a motion vector of the target block relative to the first previous frame and a motion vector of the target block relative to the first next frame.

17. The method of claim 16 , wherein

the plurality of reference frames further comprise one or more second previous frames preceding the first previous frame and one or more second next frames subsequent to the first next frame; and

determining the set of motion vectors of the target block relative to the plurality of reference frames, respectively, further comprises:

for each second previous frame, scaling the motion vector of the target block relative to the first previous frame to generate a motion vector of the target block relative to the second previous frame; and

for each second next frame, scaling the motion vector of the target block relative to the first next frame to generate a motion vector of the target block relative to the second next frame.

18. A system for performing frame rate up-conversion of video data including a sequence of image frames, comprising:

a memory configured to store the sequence of image frames; and

a video processor coupled to the memory and configured to:

determine a set of motion vectors of a target frame relative to a plurality of reference frames, wherein the target frame is to be generated and interpolated into the sequence of image frames;

perform a motion vector classification on the set of motion vectors to generate a target object map for the target frame;

project the target object map onto the plurality of reference frames to generate a plurality of reference object maps based on the set of motion vectors; and

detect an occlusion area in the target frame based on the set of motion vectors, the target object map, and the plurality of reference object maps.

19. The system of claim 18 , wherein to perform the motion vector classification on the set of motion vectors to generate the target object map for the target frame, the video processor is further configured to:

perform the motion vector classification on the set of motion vectors to detect one or more objects in the target frame;

generate the target object map to include the one or more objects; and

determine one or more relative depth values of the one or more objects in the target object map.

20. A non-transitory computer-readable storage medium configured to store instructions which, when executed by a video processor, cause the video processor to perform a process for performing frame rate up-conversion of video data including a sequence of image frames, the process comprising:

determining a set of motion vectors of a target frame relative to a plurality of reference frames, wherein the target frame is to be generated and interpolated into the sequence of image frames;

performing a motion vector classification on the set of motion vectors to generate a target object map for the target frame;

projecting the target object map onto the plurality of reference frames to generate a plurality of reference object maps based on the set of motion vectors; and

detecting an occlusion area in the target frame based on the set of motion vectors, the target object map, and the plurality of reference object maps.