Granted Patent B2

US 8,773,595 · App. 13/141,688 · Granted Jul 8, 2014

Abstract

Claims (34)

1. A method of processing video signals representing multiple images, comprising:

identifying pixels in an image that should not be included in performing motion compensation of images in the multiple images;

using a hole-filling process to replace the identified pixels with values generated from values in a background image to form a hole-filled image;

performing motion estimation on the hole-filled image to generate a motion vector field;

using the motion vector field to perform motion-compensated processing on the hole-filled image; and

combining a result of the motion-compensated processing with the identified pixel to generate an output image.

2. The method of claim 1 , wherein the background image represents background objects in the image, and the identified pixels represent foreground objects in the image.

3. The method of claim 2 , wherein the pixels represent stationary objects.

4. The method of claim 1 , wherein the pixels in the hole-filled image comprise

assigned pixel values derived from pixel values at pixel locations in the vicinity of the identified pixels.

5. The method of claim 4 , wherein performing motion estimation comprises processing identified pixel values derived from pixel values associated with a first image and processing identified pixel values derived from pixel values at corresponding locations in a second image, wherein the first and second pixel locations represent pixel pairs,

further comprising for each pixel pair wherein both pixel locations are associated with a first image layer:

calculating a first pixel similarity value between pixel values at the pixel locations identified by the pixel pair;

combining each of the first pixel similarity values to generate a first similarity metric value;

for each pixel pair wherein either of the pixel locations are associated with a second image layer:

calculating a second pixel similarity value between assigned pixel values for the pixel locations of the pixel pair that are associated with the second layer, and the pixel value at the pixel location of the pixel pair that is associated with the first image layer, if there is one; and

combining each of the second pixel similarity values to generate a second similarity metric value;

multiplying the first similarity metric value by a first constant to generate a first weighted similarity metric value, and multiplying the second similarity metric value by a second constant to generate a second weighted similarity metric value; and

adding the first weighted similarity metric value to the second weighted similarity metric value to generate a total similarity metric value.

6. The method of claim 5 , wherein one or more of the similarity metrics are calculated using a sum of absolute differences.

7. The method of claim 6 , wherein the first constant is greater than the second constant.

8. The method of claim 7 , wherein the assigned pixel values are derived from pixel values associated with the first image layer that are located in a direction that is generally perpendicular to the longitudinal direction of the foreground region.

9. The method of claim 7 , wherein the assigned pixel value is derived from pixel values associated with the first image layer that are located in a vertical direction from the pixel locations associated with the second layer in the first image.

10. The method of claim 1 , wherein dividing the image into regions and associating a first plurality of regions of the image with a first image layer and a second plurality of regions of the image with a second image layer, comprises segmentation processing.

11. The method of claim 1 , wherein the identified pixels are in an overlay layer.

12. The method of claim 11 , wherein the overlay layer represents an image selected from a group consisting of subtitles, a logo, a user-interface menu, a stationary object, and running text.

13. The method of claim 12 , wherein the method comprises part of a picture-rate up-conversion operation.

14. The method of claim 12 , wherein the motion estimation comprises identifying a spatial correspondence between image areas of consecutive images represented by the video signals.

15. The method of claim 14 , wherein identifying a spatial correspondence between image areas of consecutive images represented by the video signals comprises block matching between the consecutive images in which similarity between the block is assessed using a sum of absolute differences (SAD) metric.

16. The method of claim 15 , wherein at least some of the regions of the background image represent moving objects.

17. Data processing apparatus comprising a non-transitory computer readable storage medium containing program steps, which, when executed by the data processing apparatus, performs the method of claim 1 .

18. A display device comprising the data processing apparatus of claim 17 .

19. A computer program instantiated on a non-transitory computer readable storage medium, which, when executed by the data processing apparatus, performs the method of claim 1 .

20. A non-transitory computer program carrier, having stored thereon the computer program of claim 19 .

Assignments (9)