Method and apparatus for interpolating an image
This invention concerns the interpolation of new intermediate images within an existing set of related images, for example views of a scene captured at different times, such as a sequence of film frames or video fields, or views of a scene captured from different camera positions, either simultaneously or in a time sequence. Motion vectors established for pixels or regions in at least one existing image are used to shift those pixels or regions to the output position of the new image. The shifted pixels or regions are combined in dependence on a probability of accuracy measure for the respective motion vector. The probability of accuracy measure may be formed from displaced-field difference measurements. The occlusion of objects is determined and used in forming the probability of accuracy measure.
1. A method of interpolating in a processor a new image at an output position between first and second existing images within an existing sequence of related images, comprising the steps of:
determining motion vectors associated with pixels or regions of pixels of the first existing image in the sequence;
shifting pixels of the existing image to the output position in accordance with the or each motion vector associated therewith;
determining at least two probability of accuracy measures for each of a plurality of the said motion vectors: a first measure describing the probability that the respective associated pixel is newly revealed in the said first image of the sequence; and, a second measure describing the probability that the respective associated pixel is about to be concealed in the said first image of the sequence;
forming a weighted sum of the said shifted pixels where the weight for each pixel is proportional to a combination of the probability of accuracy measures for its respective associated motion vectors; and
creating said new image by addition of the said weighted sum of shifted pixels to a second sum of weighted shifted pixels derived from the second existing image in the sequence.
2. A method according to claim 1 in which a pixel from an existing image can contribute to two or more positions in the new image.
3. A method according to claim 1 in which the said probability of accuracy measure is derived from one or more displaced-field differences between the contributing existing image and one or more adjacent existing images.
4. A method according to claim 1 in which the said probability of accuracy measure is determined using a smooth function of a filtered displaced-field difference.
5. A method according to claim 4 in which the filtered displaced-field difference is formed from a weighted sum of a rectified displaced-field difference and an amplitude-limited, spatially low-pass filtered, rectified displaced-field difference.
6. A method according to claim 4 in which the smooth function is a negative exponential.
7. A method according to claim 1 in which the total probability of the contributions from the preceding existing field to a new image pixel is evaluated and the contribution of about-to-be-concealed pixels is reduced so that the said total does not exceed a threshold.
8. A method according to claim 1 in which the total probability of the contributions from the succeeding existing field to a new image pixel is evaluated and the contribution of newly-revealed pixels is reduced so that the said total does not exceed a threshold.
9. Apparatus for interpolating a new image at an output position between first and second existing images within an existing sequence of related images, comprising:
a motion vector input for receiving first motion vectors associated with pixels or regions of pixels of said first existing image and second motion vectors associated with pixels or regions of pixels of said second existing image;
a first vector probability determinator for determining for each of said first motion vectors a first probability of accuracy measure describing the probability that the respective associated pixel is newly revealed in the said first image of the sequence and a second probability of accuracy measure describing the probability that the respective associated pixel is about to be concealed in the said first image of the sequence;
a second vector probability determinator for determining for each of said second motion vectors a first probability of accuracy measure describing the probability that the respective associated pixel is newly revealed in the said first image of the sequence and a second probability of accuracy measure describing the probability that the respective associated pixel is about to be concealed in the said first image of the sequence;
a first motion compensator for shifting pixels of the first existing image to the output position in accordance with the or each motion vector associated therewith;
a second motion compensator for shifting pixels of the second existing image to the output position in accordance with the or each motion vector associated therewith;
a first summer for forming a first weighted sum of the said shifted pixels from said first motion compensator where the weight for each pixel is proportional to a combination of said probability of accuracy measures for its respective associated motion vectors;
a second summer for forming a second weighted sum of the said shifted pixels from said second motion compensator where the weight for each pixel is proportional to a combination of said probability of accuracy measures for its respective associated motion vectors; and
a combiner for combining shifted picture information to form said new image depending on the probability of accuracy measure for the respective motion vector by addition of said first weighted sum of shifted pixels to said second sum of weighted shifted pixels.
10. A method of interpolating a new image in a processor at an output position within an existing sequence of related images, wherein a plurality of respective motion vectors for an input pixel are used to determine a plurality of respective contributions from that input pixel to the said new image, comprising the steps of:
determining motion vectors associated with pixels or regions of pixels of a first existing image in the sequence;
shifting pixels of the said first existing image to the said output position in accordance with the or each motion vector associated therewith;
determining at least two respective probability of accuracy measures for each of a plurality of the said motion vectors: a first probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the preceding image in the sequence; and a second probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the succeeding image in the sequence;
forming a weighted sum of the said shifted pixels where the weight for each pixel is proportional to a combination of the probability of accuracy measures for its respective associated motion vectors; and
creating said new image by the addition of the said weighted sum of shifted pixels to a second sum of weighted shifted pixels derived from a second existing image in the sequence.
11. A method according to claim 10 where the said addition is weighted in proportion to the proximity of the position in the sequence of the new interpolated image to the said first and second existing images respectively.
12. A method according to claim 10 in which the said addition is weighted in dependence on the ratio of the total probabilities of vectors applicable to the pixel from the said first image in the sequence to the total probabilities of vectors applicable to the pixel from the said second image in the sequence.
13. A method according to claim 10 where a smooth function of a displaced-field difference contributes to the said probability of accuracy measure.
14. A method according to claim 13 in which the said smooth function is a negative exponential.
15. A method according to claim 10 where the said displaced-field difference is spatially filtered.
16. A method according to claim 15 in which the filtered displaced-field difference is formed from a weighted sum of a rectified displaced-field difference and an amplitude-limited, spatially low-pass filtered, rectified displaced-field difference.
17. Apparatus for interpolating a new image at an output position between first and second existing images within an existing sequence of related images, comprising:
a motion vector input for receiving first motion vectors associated with pixels or regions of pixels of said first existing image and second motion vectors associated with pixels or regions of pixels of said second existing image;
a first vector probability determinator for determining for each of said first motion vectors a first probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the preceding image in the sequence; and a second probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the succeeding image in the sequence;
a second vector probability determinator for determining for each of said second motion vectors a first probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the preceding image in the sequence; and a second probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the succeeding image in the sequence;
a first motion compensator for shifting pixels of the first existing image to the output position in accordance with the or each motion vector associated therewith;
a second motion compensator for shifting pixels of the second existing image to the output position in accordance with the or each motion vector associated therewith;
a first summer for forming a first weighted sum of the said shifted pixels from said first motion compensator where the weight for each pixel is proportional to a combination of said probability of accuracy measures for its respective associated motion vectors;
a second summer for forming a second weighted sum of the said shifted pixels from said second motion compensator where the weight for each pixel is proportional to a combination of said probability of accuracy measures for its respective associated motion vectors; and
a combiner for combining shifted picture information to form said new image depending on the probability of accuracy measure for the respective motion vector by addition of said first weighted sum of shifted pixels to said second sum of weighted shifted pixels.
18. A non-transitory computer readable medium causing a computer program containing instructions for the implementation of a method of interpolating in an image interpolator a new image at an output position between first and second existing images within an existing sequence of related images, comprising the steps of:
determining motion vectors associated with pixels or regions of pixels of the first existing image in the sequence;
shifting pixels of the existing image to the output position in accordance with the or each motion vector associated therewith;
determining at least two probability of accuracy measures for each of a plurality of the said motion vectors: a first measure describing the probability that the respective associated pixel is newly revealed in the said first image of the sequence; and, a second measure describing the probability that the respective associated pixel is about to be concealed in the said first image of the sequence;
forming a weighted sum of the said shifted pixels where the weight for each pixel is proportional to a combination of the probability of accuracy measures for its respective associated motion vectors; and
creating said new image by addition of the said weighted sum of shifted pixels to a second sum of weighted shifted pixels derived from the second existing image in the sequence.
19. A computer program product according to claim 18 in which a pixel from an existing image can contribute to two or more positions in the new image.
20. A computer program product according to claim 18 in which the said probability of accuracy measure is derived from one or more displaced-field differences between the contributing existing image and one or more adjacent existing images.
21. A computer program product according to claim 18 in which the said probability of accuracy measure is determined using a smooth function of a filtered displaced-field difference.
22. A computer program product according to claim 21 in which the filtered displaced-field difference is formed from a weighted sum of a rectified displaced-field difference and an amplitude-limited, spatially low-pass filtered, rectified displaced-field difference.
23. A computer program product according to claim 21 in which the smooth function is a negative exponential.
24. A computer program product according to claim 18 in which the total probability of the contributions from the preceding existing field to a new image pixel is evaluated and the contribution of about-to-be-concealed pixels is reduced so that the said total does not exceed a threshold.
25. A computer program product according to claim 18 in which the total probability of the contributions from the succeeding existing field to a new image pixel is evaluated and the contribution of newly-revealed pixels is reduced so that the said total does not exceed a threshold.
26. A non-transitory computer program product containing instructions for the implementation of a method of interpolating a new image in an image interpolator at an output position within an existing sequence of related images, wherein a plurality of respective motion vectors for an input pixel are used to determine a plurality of respective contributions from that input pixel to the said new image, comprising the steps of:
determining motion vectors associated with pixels or regions of pixels of a first existing image in the sequence;
shifting pixels of the said first existing image to the said output position in accordance with the or each motion vector associated therewith;
determining at least two respective probability of accuracy measures for each of a plurality of the said motion vectors: a first probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the preceding image in the sequence; and a second probability of accuracy measure that is a function of a displaced-field difference between the pixel associated with the said vector and a corresponding pixel in the succeeding image in the sequence;
forming a weighted sum of the said shifted pixels where the weight for each pixel is proportional to a combination of the probability of accuracy measures for its respective associated motion vectors; and
creating said new image by the addition of the said weighted sum of shifted pixels to a second sum of weighted shifted pixels derived from a second existing image in the sequence.
27. A computer program product according to claim 26 where the said addition is weighted in proportion to the proximity of the position in the sequence of the new interpolated image to the said first and second existing images respectively.
28. A computer program product according to claim 26 in which the said addition is weighted in dependence on the ratio of the total probabilities of vectors applicable to the pixel from the said first image in the sequence to the total probabilities of vectors applicable to the pixel from the said second image in the sequence.
29. A method according to claim 26 where a smooth function of a displaced-field difference contributes to the said probability of accuracy measure.
30. A method according to claim 26 in which the said smooth function is a negative exponential.
31. A method according to claim 26 where the said displaced-field difference is spatially filtered.
32. A method according to claim 31 in which the filtered displaced-field difference is formed from a weighted sum of a rectified displaced-field difference and an amplitude-limited, spatially low-pass filtered, rectified displaced-field difference.