Cold matching by automatic content recognition
A method for cold matching by automatic content recognition includes receiving, at data processing hardware, broadcast fingerprints indicative of broadcast frames of a broadcast media stream. The method also includes receiving, at the data processing hardware, media device fingerprints indicative of media device frames of a media device stream at a corresponding media device. The method further includes determining, by the data processing hardware, that a search block comprising a plurality of media device fingerprints corresponding to sequential media device frames of the media device stream includes at least one media device fingerprint having a match correlation to at least one broadcast fingerprint. The match correlation is based on a Pearson correlation coefficient. The method also includes identifying, by the data processing hardware, the at least one media device fingerprint having the match correlation to the at least one broadcast fingerprint as corresponding to a match media device frame.
1. A method of identifying media provided for viewing at a media device, the method comprising:
receiving, at data processing hardware, broadcast fingerprints indicative of sequential broadcast frames of a broadcast media stream, the sequential broadcast frames corresponding to a first time period of the broadcast media stream;
compiling, by the data processing hardware, a search block of media device fingerprints indicative of sequential media device frames of a media device stream provided for viewing at the media device, the sequential media device frames of the search block corresponding to a second time period of the media device stream, the second time period being shorter than the first time period;
comparing, by the data processing hardware, (i) the search block of the media device fingerprints indicative of the sequential media device frames corresponding with the second time period of the media device stream with (ii) the broadcast fingerprints indicative of the sequential broadcast frames corresponding to the first time period of the broadcast media stream;
establishing, by the data processing hardware, based on the comparing, a match correlation between at least one media device fingerprint of the search block and at least one of the broadcast fingerprints, wherein establishing the match correlation is based on a Pearson correlation coefficient, wherein the at least one media device fingerprint corresponds with at least one of the media device frames, and wherein the at least one broadcast fingerprint corresponds with at least one of the broadcast frames; and
determining, by the data processing hardware, based on the established match correlation, that the at least one media device frame has metadata corresponding with the at least one broadcast frame, the metadata comprising a broadcast channel identification and frame location.
2. The method of claim 1 , further comprising:
determining, by the data processing hardware, whether the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint; and
when the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint, (i) determining, by the data processing hardware, a set of broadcast fingerprints corresponding to the at least one media device fingerprint of the search block, each broadcast fingerprint of the set of broadcast fingerprints being broadcast fingerprints of sequentially adjacent broadcast frames of the broadcast media stream and (ii) determining, by the data processing hardware, the match correlation as a Pearson correlation coefficient between the search block and the set of broadcast fingerprints.
3. The method of claim 2 , further comprising determining, by the data processing hardware, a match media device frame as the media device frame corresponding to the media device fingerprint of the at least one media device fingerprint of the search block having the greatest Pearson correlation coefficient with a match broadcast fingerprint of the set of broadcast fingerprints.
4. The method of claim 1 , wherein each fingerprint represents at least one pixel value of the frame of the corresponding media stream.
5. The method of claim 1 , wherein each fingerprint is an average pixel value representing a sum of a grayscale value of a corresponding frame.
6. The method of claim 1 , wherein each fingerprint represents more than one average pixel value of a sub-frame within a corresponding frame.
7. The method of claim 6 , wherein each fingerprint of the corresponding frame is represented as a sixteen integer vector corresponding to each average pixel value of sixteen sub-frames, the sixteen sub-frames defining a four by four array of the corresponding frame.
8. The method of claim 1 , wherein comparing the search block of the media device fingerprints with the broadcast fingerprints comprises comparing the search block to a database structure defined by a plurality of broadcast fingerprints wherein the database structure is a binary search tree.
9. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations including:
receiving broadcast fingerprints indicative of sequential broadcast frames of a broadcast media stream, the sequential broadcast frames corresponding to a first time period of the broadcast media stream,
compiling a search block of media device fingerprints indicative of sequential media device frames of a media device stream provided for viewing at a media device, the sequential media device frames of the search block corresponding to a second time period of the media device stream, the second time period being shorter than the first time period,
comparing (i) the search block of the media device fingerprints indicative of the sequential media device frames corresponding with the second time period of the media device stream with (ii) the broadcast fingerprints indicative of the sequential broadcast frames corresponding to the first time period of the broadcast media stream,
establishing, based on the comparing, a match correlation between at least one media device fingerprint of the search block and at least one of the broadcast fingerprints, wherein establishing the match correlation is based on a Pearson correlation coefficient, wherein the at least one media device fingerprint corresponds with at least one of the media device frames, and wherein the at least one broadcast fingerprint corresponds with at least one of the broadcast frames, and
determining, based on the established match correlation, that the at least one media device frame has metadata corresponding with the at least one broadcast frame, the metadata comprising a broadcast channel identification and frame location.
10. The system of claim 9 , wherein the operations further include:
determining whether the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint, and
when the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint, (i) determining a set of broadcast fingerprints corresponding to the at least one media device fingerprint of the search block, each broadcast fingerprint of the set of broadcast fingerprints being broadcast fingerprints of sequentially adjacent broadcast frames of the broadcast media stream and (ii) determining the match correlation as a Pearson correlation coefficient between the search block and the set of broadcast fingerprints.
11. The system of claim 10 , wherein the operations further include determining a match media device frame as the media device frame corresponding to the media device fingerprint of the at least one media device fingerprint of the search block having the greatest Pearson correlation coefficient with a match broadcast fingerprint of the set of broadcast fingerprints.
12. The system of claim 9 , wherein each fingerprint represents at least one pixel value of the frame of the corresponding media stream.
13. The system of claim 9 , wherein each fingerprint represents more than one average pixel value of a sub-frame within a corresponding frame.
14. The system of claim 13 , wherein each fingerprint of the corresponding frame is represented as a sixteen integer vector corresponding to each average pixel value of sixteen sub-frames, the sixteen sub-frames defining a four by four array of the corresponding frame.
15. The system of claim 9 , wherein comparing the search block of the media device fingerprints with the broadcast fingerprints comprises comparing the search block to a database structure defined by a plurality of broadcast fingerprints wherein the database structure is a binary search tree.
16. A non-transitory memory storing program instructions executable by a processor to carry out operations including:
receiving broadcast fingerprints indicative of sequential broadcast frames of a broadcast media stream, the sequential broadcast frames corresponding to a first time period of the broadcast media stream,
compiling a search block of media device fingerprints indicative of sequential media device frames of a media device stream provided for viewing at a media device, the sequential media device frames of the search block corresponding to a second time period of the media device stream, the second time period being shorter than the first time period,
comparing (i) the search block of the media device fingerprints indicative of the sequential media device frames corresponding with the second time period of the media device stream with (ii) the broadcast fingerprints indicative of the sequential broadcast frames corresponding to the first time period of the broadcast media stream,
establishing, based on the comparing, a match correlation between at least one media device fingerprint of the search block and at least one of the broadcast fingerprints, wherein establishing the match correlation is based on a Pearson correlation coefficient, wherein the at least one media device fingerprint corresponds with at least one of the media device frames, and wherein the at least one broadcast fingerprint corresponds with at least one of the broadcast frames, and
determining, based on the established match correlation, that the at least one media device frame has metadata corresponding with the at least one broadcast frame, the metadata comprising a broadcast channel identification and frame location.
17. The non-transitory memory of claim 16 , wherein the operations further include:
determining whether the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint, and
when the search block includes at least one media device fingerprint having the match correlation to more than one broadcast fingerprint, (i) determining a set of broadcast fingerprints corresponding to the at least one media device fingerprint of the search block, each broadcast fingerprint of the set of broadcast fingerprints being broadcast fingerprints of sequentially adjacent broadcast frames of the broadcast media stream and (ii) determining the match correlation as a Pearson correlation coefficient between the search block and the set of broadcast fingerprints.
18. The non-transitory memory of claim 16 , wherein the operations further include determining a match media device frame as the media device frame corresponding to the media device fingerprint of the at least one media device fingerprint of the search block having the greatest Pearson correlation coefficient with a match broadcast fingerprint of the set of broadcast fingerprints.
19. The non-transitory memory of claim 16 , wherein comparing the search block of the media device fingerprints with the broadcast fingerprints comprises comparing the search block to a database structure defined by a plurality of broadcast fingerprints wherein the database structure is a binary search tree.