Extracting features of video and audio signal content to provide reliable identification of the signals
Signatures that can be used to identify video and audio content are generated from the content by generating measures of dissimilarity between features of corresponding groups of pixels in frames of video content and by generating low-resolution time-frequency representations of audio segments. The signatures are generated by applying a hash function to intermediate values derived from the measures of dissimilarity and to the low-resolution time-frequency representations. The generated signatures may be used in a variety of applications such as restoring synchronization between video and audio content streams and identifying copies of original video and audio content. The generated signatures can provide reliable identifications despite intentional and unintentional modifications to the content.
1. A method performed in a device for generating a signature that identifies content of a video signal, wherein the method comprises:
the device extracting a first set of one or more features from data representing pixels in images conveyed by a first video frame within a sequence of video frames of the video signal and calculating first values representing the first set of features;
the device extracting a second set of one or more features from data representing pixels in images conveyed by a second video frame within the sequence of video frames of the video signal, wherein each feature in the second set of features has a corresponding feature in the first set of features, and calculating second values representing the second set of features, wherein the second value that represents a respective feature in the second set of features corresponds to the first value that represents the feature in the first set of features corresponding to the respective feature;
the device generating an array of intermediate values, wherein each intermediate value in the array represents one or more measures of dissimilarity between second values and the corresponding first values; and
the device generating the signature that identifies content of the video signal by applying a hash function to the intermediate values wherein the applying comprises projecting the intermediate values onto a set of random vectors, and wherein:
each component of the signature is derived from the projection of the intermediate values onto a respective random vector;
each respective vector in the set of random vectors has vector elements with values that are obtained from a difference between uniformly distributed random variables within a range from zero to one and an average of the uniformly distributed random variables for all vector elements of the respective vector;
the projection of the intermediate values onto a respective random vector is obtained from an inner product of the intermediate values with the vector elements of the respective vector; and
each component of the signature has either a first value if the projection of the intermediate values onto a corresponding random vector is greater than a threshold or has a second value if the projection of the intermediate values onto the corresponding random vector is less than a threshold.
2. The method of claim 1 that comprises the device extracting the first set of features and the second features from groups of pixels, each group of pixels representing a respective region of one of the images conveyed by the first and second video frames.
3. The method of claim 1 , wherein the first and second sets of features represent intensities of the pixels and a respective measure of dissimilarity is an absolute difference between the second values and the corresponding first values.
4. The method of claim 3 that comprises:
the device grouping a plurality of the measures of dissimilarity into regions of a difference image; and
the device generating each intermediate value to represent an average of the measures of dissimilarity in a respective region.
5. The method of claim 1 that comprises:
the device determining horizontal and vertical resolutions of the images conveyed by the first and second video frames;
the device selecting a factor in response to the resolutions; and
the device down sampling the data, the measures of dissimilarity or the intermediate values by an amount equal to the factor to reduce vertical and horizontal resolutions of the array of intermediate values.
6. An apparatus for generating a signature that identifies content of a video signal, wherein the apparatus comprises:
means for extracting a first set of one or more features from data representing pixels in images conveyed by a first video frame within a sequence of video frames of the video signal and calculating first values representing the first set of features;
means for extracting a second set of one or more features from data representing pixels in images conveyed by a second video frame within the sequence of video frames of the video signal, wherein each feature in the second set of features has a corresponding feature in the first set of features, and calculating second values representing the second set of features, wherein the second value that represents a respective feature in the second set of features corresponds to the first value that represents the feature in the first set of features corresponding to the respective feature;
means for generating an array of intermediate values, wherein each intermediate value in the array represents one or more measures of dissimilarity between second values and the corresponding first values; and
means for generating the signature that identifies content of the video signal by applying a hash function to the intermediate values wherein the applying comprises projecting the intermediate values onto a set of random vectors, and wherein:
each component of the signature is derived from the projection of the intermediate values onto a respective random vector;
each respective vector in the set of random vectors has vector elements with values that are obtained from a difference between uniformly distributed random variables within a range from zero to one and an average of the uniformly distributed random variables for all vector elements of the respective vector;
the projection of the intermediate values onto a respective random vector is obtained from an inner product of the intermediate values with the vector elements of the respective vector; and
each component of the signature has either a first value if the projection of the intermediate values onto a corresponding random vector is greater than a threshold or has a second value if the projection of the intermediate values onto the corresponding random vector is less than a threshold.
7. A storage medium that records a program of instructions that is executable by a device to perform a method for generating a signature that identifies content of a video signal, wherein the method comprises:
extracting a first set of one or more features from data representing pixels in images conveyed by a first video frame within a sequence of video frames of the video signal and calculating first values representing the first set of features;
extracting a second set of one or more features from data representing pixels in images conveyed by a second video frame within the sequence of video frames of the video signal, wherein each feature in the second set of features has a corresponding feature in the first set of features, and calculating second values representing the second set of features, wherein the second value that represents a respective feature in the second set of features corresponds to the first value that represents the feature in the first set of features corresponding to the respective feature;
generating an array of intermediate values, wherein each intermediate value in the array represents one or more measures of dissimilarity between second values and the corresponding first values; and
generating the signature that identifies content of the video signal by applying a hash function to the intermediate values wherein the applying comprises projecting the intermediate values onto a set of random vectors, and wherein:
each component of the signature is derived from the projection of the intermediate values onto a respective random vector;
each respective vector in the set of random vectors has vector elements with values that are obtained from a difference between uniformly distributed random variables within a range from zero to one and an average of the uniformly distributed random variables for all vector elements of the respective vector;
the projection of the intermediate values onto a respective random vector is obtained from an inner product of the intermediate values with the vector elements of the respective vector; and
each component of the signature has either a first value if the projection of the intermediate values onto a corresponding random vector is greater than a threshold or has a second value if the projection of the intermediate values onto the corresponding random vector is less than a threshold.
8. The storage medium of claim 7 , wherein the method comprises extracting the first set of features and the second features from groups of pixels, each group of pixels representing a respective region of one of the images conveyed by the first and second video frames.
9. The storage medium of claim 7 , wherein the first and second sets of features represent intensities of the pixels and a respective measure of dissimilarity is an absolute difference between the second values and the corresponding first values.
10. The storage medium of claim 9 , wherein the method comprises:
grouping a plurality of the measures of dissimilarity into regions of a difference image; and
generating each intermediate value to represent an average of the measures of dissimilarity in a respective region.
11. The storage medium of claim 7 , wherein the method comprises:
determining horizontal and vertical resolutions of the images conveyed by the first and second video frames;
selecting a factor in response to the resolutions; and
down sampling the data, the measures of dissimilarity or the intermediate values by an amount equal to the factor to reduce vertical and horizontal resolutions of the array of intermediate values.
12. The apparatus of claim 6 that comprises means for extracting the first set of features and the second features from groups of pixels, each group of pixels representing a respective region of one of the images conveyed by the first and second video frames.
13. The apparatus of claim 6 , wherein the first and second sets of features represent intensities of the pixels and a respective measure of dissimilarity is an absolute difference between the second values and the corresponding first values.
14. The apparatus of claim 13 that comprises:
means for grouping a plurality of the measures of dissimilarity into regions of a difference image; and
means for generating each intermediate value to represent an average of the measures of dissimilarity in a respective region.
15. The apparatus of claim 6 that comprises:
means for determining horizontal and vertical resolutions of the images conveyed by the first and second video frames;
means for selecting a factor in response to the resolutions; and
means for down sampling the data, the measures of dissimilarity or the intermediate values by an amount equal to the factor to reduce vertical and horizontal resolutions of the array of intermediate values.
16. The apparatus of claim 6 , wherein the threshold is equal to a median of the projections of intermediate values onto the set of random vectors.
17. The method of claim 1 , wherein the threshold is equal to a median of the projections of intermediate values onto the set of random vectors.
18. The storage medium of claim 7 , wherein the threshold is equal to a median of the projections of intermediate values onto the set of random vectors.