Audio-based synchronization to media
An audio channel of a time-based media presentation provides a basis for synchronizing to the presentation across a variety of platforms independent of when and where the presentation is being viewed. By pre-processing the media into a series of non-unique hashes, and similarly processing an audio stream of the media captured at a client device, a comparison can be made that yields an accurate time offset within the presentation. The comparison may usefully be performed over a data network using a server that hosts data from the pre-processed media, and a variety of applications may be deployed on the client device based on the resulting synchronization.
1. A method comprising:
receiving an audio portion of a time-based media presentation with a microphone of a client device;
sampling the audio on the client device to obtain a sequence of digital samples of the audio portion;
processing the sequence of digital samples on the client device to provide a plurality of hashes, each one of the plurality of hashes including a plurality of bits, each one of the plurality of hashes providing a non-unique representation of a segment of the audio portion, and each one of the plurality of hashes having a known relative time offset to each other one of the plurality of hashes;
transmitting the plurality of hashes and a unique identifier for the time-based media presentation to a server, wherein the unique identifier explicitly identifies the time-based media presentation to the server;
receiving from the server a time offset indicative of a current time offset within the time-based media presentation; and
synchronizing an application on the client device to the time-based media presentation based upon the time offset.
2. The method of claim 1 wherein the client device includes at least one of a mobile device, a cellular phone, a laptop computer, a notebook computer, and a netbook.
3. The method of claim 1 wherein the time-based media presentation includes one or more of a movie, a sports event, and a television broadcast.
4. The method of claim 1 wherein adjacent ones of the plurality of hashes are calculated from overlapping windows of the sequence of digital samples.
5. The method of claim 1 wherein processing includes filtering the sequence of digital samples with a low pass filter to provide a filtered output and transforming the filtered output with a discrete Fourier transform.
6. The method of claim 1 wherein processing includes normalizing a magnitude of the sequence of digital samples.
7. The method of claim 1 further comprising, on the server,
determining an allowable bit error for the plurality of hashes;
identifying for each one of the plurality of hashes a set of candidate hashes with a number of bitwise variations from the one of the plurality of hashes no greater than the allowable bit error;
locating any candidate time offsets in the time-based media presentation corresponding to each set of candidate hashes for each one of the plurality of hashes;
updating scores for the candidate time offsets corresponding to each set of candidate hashes;
selecting one of the candidate time offsets having a best one of the scores as the time offset within the time-based media presentation; and
transmitting the time offset to the client device.
8. The method of claim 7 wherein determining the allowable bit error includes receiving the allowable bit error from the client device.
9. The method of claim 8 wherein each one of the plurality of hashes consists of thirty two bits and the allowable bit error is eight bits.
10. The method of claim 7 wherein identifying a set of candidate hashes includes providing a binary tree of all possible values for the hash and traversing the binary tree in a manner that excludes branches for binary values that exceed the allowable bit error for the hash.
11. The method of claim 7 further comprising conditionally transmitting the time offset only when the best one of the scores exceeds a predetermined threshold.
12. The method of claim 1 further comprising transmitting supplemental information to the server including a hash sequence number that identifies an order of the plurality of hashes relative to one another.
13. The method of claim 1 wherein synchronizing includes displaying an indicator on the client device that indicates a synchronization with the time-based media presentation.
14. The method of claim 1 further comprising rendering additional content on the client device under control of the application, the additional content synchronized to the time-based media presentation.
15. The method of claim 14 wherein rendering additional content includes rendering one or more of a supplemental video stream, contextual information, advertising, and interactive content.
16. A computer program product for audio-based synchronization, the computer program product comprising computer executable code embodied on a non-transitory computer readable medium that, when executing on one or more computing devices, performs the steps of:
receiving an audio portion of a time-based media presentation with a microphone of a client device;
sampling the audio on the client device to obtain a sequence of digital samples of the audio portion;
processing the sequence of digital samples on the client device to provide a plurality of hashes, each one of the plurality of hashes including a plurality of bits, each one of the plurality of hashes providing a non-unique representation of a segment of the audio portion, and each one of the plurality of hashes having a known relative time offset to each other one of the plurality of hashes;
transmitting the plurality of hashes and a unique identifier for the time-based media presentation to a server, wherein the unique identifier explicitly identifies the time-based media presentation to the server;
receiving from the server a time offset indicative of a current time offset within the time-based media presentation; and
synchronizing an application on the client device to the time-based media presentation based upon the time offset.
17. The computer program product of claim 16 wherein the client device includes at least one of a mobile device, a cellular phone, a laptop computer, a notebook computer, and a netbook.
18. The computer program product of claim 16 wherein the time-based media presentation includes one or more of a movie, a sports event, and a television broadcast.
19. The computer program product of claim 16 wherein adjacent ones of the plurality of hashes are calculated from overlapping windows of the sequence of digital samples.