Scene frame matching for automatic content recognition

1. A method comprising:

receiving, at data processing hardware, broadcast fingerprints indicative of broadcast frames of a broadcast media stream, the broadcast media stream comprising scenes, each scene comprising a plurality of broadcast frames ranging from an initial frame having a corresponding initial fingerprint to a last frame having a corresponding last fingerprint;

for each of at least two sequentially adjacent pairs of broadcast frames:

determining, by the data processing hardware, a statistical correlation coefficient between the respective broadcast fingerprints;

determining, by the data processing hardware, whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying, by the data processing hardware, a first scene as ending at a first broadcast frame of the pair of broadcast frames and a second scene as starting at a second broadcast frame of the pair of broadcast frames;

for each of at least two identified scenes, determining, by the data processing hardware, a respective scene centroid, each scene centroid being a mean of fingerprints of the plurality of broadcast frames of a respective scene, the mean of fingerprints determined by an average of pixel values corresponding to the fingerprints of the respective scene;

comparing, by the data processing hardware, each of at least two of the scene centroids to a target fingerprint of a corresponding single target frame of the broadcast media stream; and

based on comparisons, determining, by the data processing hardware, that the single target frame belongs to a particular one of the scenes represented by the at least two of the scene centroids, such that the single target frame is one of the plurality of broadcast frames that make up the particular scene.

2. The method of claim 1 ,

wherein comparing each of the at least two of the scene centroids to the target fingerprint of the corresponding single target frame of the broadcast media stream comprises for each of the at least two scene centroids, determining a distance between the respective scene centroid and the target fingerprint, and

wherein determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on the comparisons comprises determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on a scene centroid representing the particular one of the scenes having the shortest distance to the target fingerprint of the single target frame.

3. The method of claim 2 , further comprising:

for a broadcast frame of the particular one of the scenes:

determining, by the data processing hardware, a statistical correlation coefficient between the respective broadcast fingerprint and the target fingerprint;

determining, by the data processing hardware, whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying, by the data processing hardware, the broadcast frame as being the single target frame.

4. The method of claim 1 , wherein each of the determined scene centroids represents a time centroid representing a range from an initial time of the initial broadcast fingerprint to a last time of the last broadcast fingerprint.

5. The method of claim 1 , wherein each fingerprint represents a 16 integer vector corresponding to average pixel values of sixteen sub-frames, the sixteen sub-frames defining a four by four array of the corresponding frame.

6. The method of claim 1 , wherein the statistical correlation coefficient is a Pearson correlation coefficient.

7. The method of claim 1 , wherein the statistical correlation coefficient is a Kendall correlation coefficient.

8. The method of claim 1 , wherein the statistical correlation coefficient is a Spearman correlation coefficient.

9. A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:

receiving broadcast fingerprints indicative of broadcast frames of a broadcast media stream, the broadcast media stream comprising scenes, each scene comprising a plurality of broadcast frames ranging from an initial frame having a corresponding initial fingerprint to a last frame having a corresponding last fingerprint;

for each of at least two sequentially adjacent pairs of broadcast frames:

determining a statistical correlation coefficient between the respective broadcast fingerprints;

determining whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying a first scene as ending at a first broadcast frame of the pair of broadcast frames and a second scene as starting at a second broadcast frame of the pair of broadcast frames;

for each of at least two identified scenes, determining a respective scene centroid, each scene centroid being a mean of fingerprints of the plurality of broadcast frames of a respective scene, the mean of fingerprints determined by an average of pixel values corresponding to the fingerprints of the respective scene;

comparing each of at least two of the scene centroids to a target fingerprint of a corresponding single target frame of the broadcast media stream; and

based on the comparisons, determining that the single target frame belongs to a particular one of the scenes represented by the at least two of the scene centroids, such that the single target frame is one of the plurality of broadcast frames that make up the particular scene.

10. The system of claim 9 ,

wherein comparing each of the at least two of the scene centroids to the target fingerprint of the corresponding single target frame of the broadcast media stream comprises for each of the at least two scene centroids, determining a distance between the respective scene centroid and the target fingerprint, and

wherein determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on the comparisons comprises determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on a scene centroid representing the particular one of the scenes having the shortest distance to the target fingerprint of the single target frame.

11. The system of claim 9 , wherein the operations further comprise:

for a broadcast frame of the particular one of the scenes:

determining a statistical correlation coefficient between the respective broadcast fingerprint and the target fingerprint;

determining whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying the broadcast frame as being the single target frame.

12. The system of claim 9 , wherein each of the determined scene centroids represents a time centroid representing a range from an initial time of the initial broadcast fingerprint to a last time of the last broadcast fingerprint.

13. The system of claim 9 , wherein each fingerprint represents a 16 integer vector corresponding to average pixel values of sixteen sub-frames, the sixteen sub-frames defining a four by four array of the corresponding frame.

14. The system of claim 9 , wherein the statistical correlation coefficient is selected from the group consisting of a Pearson correlation coefficient, a Kendall correlation coefficient, and a Spearman correlation coefficient.

15. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a processor, cause performance of a set of acts comprising:

receiving broadcast fingerprints indicative of broadcast frames of a broadcast media stream, the broadcast media stream comprising scenes, each scene comprising a plurality of broadcast frames ranging from an initial frame having a corresponding initial fingerprint to a last frame having a corresponding last fingerprint;

for each of at least two sequentially adjacent pairs of broadcast frames:

determining a statistical correlation coefficient between the respective broadcast fingerprints;

determining whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying a first scene as ending at a first broadcast frame of the pair of broadcast frames and a second scene as starting at a second broadcast frame of the pair of broadcast frames;

for each of at least two identified scenes, determining a respective scene centroid, each scene centroid being a mean of fingerprints of the plurality of broadcast frames of a respective scene, the mean of fingerprints determined by an average of pixel values corresponding to the fingerprints of the respective scene;

comparing each of at least two of the scene centroids to a target fingerprint of a corresponding single target frame of the broadcast media stream; and

based on the comparisons, determining that the single target frame belongs to a particular one of the scenes represented by the at least two of the scene centroids, such that that single target frame is one of the plurality of broadcast frames that make up the particular scene.

16. The non-transitory computer readable medium of claim 15 ,

wherein comparing each of the at least two of the scene centroids to the target fingerprint of the corresponding single target frame of the broadcast media stream comprises for each of the at least two scene centroids, determining a distance between the respective scene centroid and the target fingerprint, and

wherein determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on the comparisons comprises determining that the single target frame belongs to the particular one of the scenes represented by the at least two of the scene centroids based on a scene centroid representing the particular one of the scenes having the shortest distance to the target fingerprint of the single target frame.

17. The non-transitory computer readable medium of claim 15 , the set of acts further comprising:

for a broadcast frame of the particular one of the scenes:

determining a statistical correlation coefficient between the respective broadcast fingerprint and the target fingerprint;

determining whether the statistical correlation coefficient satisfies a correlation threshold; and

when the statistical correlation coefficient satisfies the correlation threshold, identifying the broadcast frame as being the single target frame.

18. The non-transitory computer readable medium of claim 15 , wherein each of the determined scene centroids represents a time centroid representing a range from an initial time of the initial broadcast fingerprint to a last time of the last broadcast fingerprint.

19. The non-transitory computer readable medium of claim 15 , wherein each fingerprint represents a 16 integer vector corresponding to average pixel values of sixteen sub-frames, the sixteen sub-frames defining a four by four array of the corresponding frame.

20. The non-transitory computer readable medium of claim 15 , wherein the statistical correlation coefficient is selected from the group consisting of a Pearson correlation coefficient, a Kendall correlation coefficient, and a Spearman correlation coefficient.