Media content identification on mobile devices
A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.
1. A computer-implemented method comprising:
encountering, by a computing device, a first trigger to transmit a first query fingerprint to a server device;
based on the encountering of the first trigger: (i) incrementing, by the computing device, a counter and (ii) transmitting, by the computing device to the server device, the first query fingerprint for comparison with reference fingerprints of a reference database;
after encountering the first trigger, encountering, by the computing device, a second trigger to transmit a second query fingerprint to the server device;
determining, by the computing device, that a number stored by the counter satisfies a threshold condition, wherein the number is indicative of a successive number of encountered triggers without matching a query fingerprint to any reference fingerprints of a set of reference fingerprints that are stored in a memory of the computing device; and
based on the determining that the number satisfies the threshold condition, forgoing transmission of the second query fingerprint to the server device.
2. The computer-implemented method of claim 1 , further comprising incrementing another counter based on the encountering of the second trigger and the determining that the number satisfies the threshold condition.
3. The computer-implemented method of claim 2 , further comprising:
after encountering the second trigger, encountering a third trigger to transmit a third query fingerprint to the server device;
determining that a number stored by the other counter satisfies another threshold condition; and
based on the determining the number stored by the other counter satisfies the other threshold condition, transmitting the third query fingerprint to the server device.
4. The computer-implemented method of claim 3 , further comprising:
determining that the third query fingerprint matches a reference fingerprint of the reference database; and
based on the determining that the third query fingerprint matches the reference fingerprint, resetting the counter and the other counter.
5. The computer-implemented method of claim 1 , further comprising:
after encountering the second trigger, obtaining a third query fingerprint;
determining that the third query fingerprint matches a reference fingerprint of a set of reference fingerprints that are stored in a memory of the computing device; and
based on the determining that the third query fingerprint matches the reference fingerprint, resetting the counter.
6. The computer-implemented method of claim 1 , wherein encountering the first trigger comprises:
obtaining a fragment of media content, wherein the fragment of media content is a representation of a sequence of media content that is presented by a playback device;
generating the first query fingerprint using the fragment of media content; and
determining that the first query fingerprint does not match any reference fingerprints of the set of reference fingerprints.
7. The computer-implemented method of claim 6 , wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a microphone, wherein the fragment of media content comprises an audio fragment, and wherein the first query fingerprint comprises an audio fingerprint.
8. The computer-implemented method of claim 6 , wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a camera, wherein the fragment of media content comprises a video fragment, and wherein the first query fingerprint comprises a video fingerprint.
9. The computer-implemented method of claim 6 , wherein the playback device comprises a television.
10. A computing device comprising:
one or more processors; and
a non-transitory computer-readable medium having stored therein instructions that are executable by the one or more processors to cause the computing device to perform functions comprising:
encountering a first trigger to transmit a first query fingerprint to a server device,
based on the encountering of the first trigger: (i) incrementing a counter and (ii) transmitting, to the server device, the first query fingerprint for comparison with reference fingerprints of a reference database,
after encountering the first trigger, encountering a second trigger to transmit a second query fingerprint to the server device,
determining that a number stored by the counter satisfies a threshold condition, wherein the number is indicative of a successive number of encountered triggers without matching a query fingerprint to any reference fingerprints of a set of reference fingerprints that are stored in a memory of the computing device, and
based on the determining that the number satisfies the threshold condition, forgoing transmission of the second query fingerprint to the server device.
11. The computing device of claim 10 , wherein the functions further comprise incrementing another counter based on the encountering of the second trigger and the determining that the number satisfies the threshold condition.
12. The computing device of claim 11 , wherein the functions further comprise:
after encountering the second trigger, encountering a third trigger to transmit a third query fingerprint to the server device;
determining that a number stored by the other counter satisfies a threshold condition; and
based on the determining the number stored by the other counter satisfies the threshold condition, transmitting the third query fingerprint to the server device.
13. The computing device of claim 10 , wherein encountering the first trigger comprises:
obtaining a fragment of media content, wherein the fragment of media content is a representation of a sequence of media content that is presented by a playback device;
generating the first query fingerprint using the fragment of media content; and
determining that the first query fingerprint does not match any reference fingerprints of the set of reference fingerprints.
14. The computing device of claim 13 , further comprising a microphone, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using the microphone, wherein the fragment of media content comprises an audio fragment, and wherein the first query fingerprint comprises an audio fingerprint.
15. The computing device of claim 13 , further comprising a camera, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using the camera, wherein the fragment of media content comprises a video fragment, and wherein the first query fingerprint comprises a video fingerprint.
16. A non-transitory computer-readable medium having stored therein instructions that are executable by one or more processors to cause a computing device to perform functions comprising:
encountering a first trigger to transmit a first query fingerprint to a server device;
based on the encountering of the first trigger: (i) incrementing a counter and (ii) transmitting, to the server device, the first query fingerprint for comparison with reference fingerprints of a reference database;
after encountering the first trigger, encountering a second trigger to transmit a second query fingerprint to the server device;
determining that a number stored by the counter satisfies a threshold condition, wherein the number is indicative of a successive number of encountered triggers without matching a query fingerprint to any reference fingerprints of a set of reference fingerprints that are stored in a memory of the computing device; and
based on the determining that the number satisfies the threshold condition, forgoing transmission of the second query fingerprint to the server device.
17. The non-transitory computer-readable medium of claim 16 , wherein encountering the first trigger comprises:
obtaining a fragment of media content, wherein the fragment of media content is a representation of a sequence of media content that is presented by a playback device;
generating the first query fingerprint using the fragment of media content; and
determining that the first query fingerprint does not match any reference fingerprints of the set of reference fingerprints.
18. The non-transitory computer-readable medium of claim 17 , wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a microphone, wherein the fragment of media content comprises an audio fragment, and wherein the first query fingerprint comprises an audio fingerprint.