Content-modification system with jitter effect mitigation feature
One high-level aspect of a content-modification system and related methods may involve facilitating content modification by a content-presentation device of received broadcast content in a controlled manner, while under circumstances in which some events that can impact timing may be unpredictable. In particular, certain operations by a content-presentation device may involve matching received content with specific expected content as determined by one or another component of the content-modification system, in order to confirm proper conditions are met for the content-presentation device to proceed with, or continue, content-modification operations. It can happen the matching procedure becomes subject or susceptible to timing irregularities, or jitter. In some instances, jitter may impact the ability to derive the benefits of content modification. Accordingly, example embodiments herein are directed to systems and method for compensation and/or mitigating the effects of jitter.
1. A method carried out by a content-presentation device of a content-modification system that includes a content-distribution system, the method comprising:
making a first determination of an upcoming content-modification opportunity in content data received by the content-presentation device on an identified channel from the content-distribution system, wherein identification of the channel is based on a match between first reference fingerprint data representing a first transmitted portion of the content data transmitted by the content-distribution system and first query fingerprint data representing at least a portion of the received content data, wherein the received content data comprises a sequence of content frames received at a source frame rate, and wherein the first query fingerprint data comprise a sequence of first query fingerprints that are generated by the content-presentation device from a first sub-sequence of content frames by sampling a first received portion of the sequence of content frames at a first frame sampling rate that is less than the source frame rate;
receiving second reference fingerprint data representing a second transmitted portion of the content data transmitted by the content-distribution system, the second reference fingerprint data comprising a sequence of reference fingerprints received at the source frame rate, wherein the second transmitted portion follows the first transmitted portion;
in response to receiving the second reference fingerprint data: (i) switching from the first frame sampling rate to a second, higher frame sampling rate, wherein the second frame sampling rate is no greater than the source frame rate, (ii) obtaining a second sub-sequence of content frames by sampling a second received portion of the received sequence of content frames at the second frame sampling rate, wherein the second received portion follows the first received portion, (iii) generating second query fingerprint data from the second sub-sequence of content frames, the second query fingerprint data comprising a sequence of second query fingerprints corresponding to the second sub-sequence of content frames, and (iv) making a second determination of whether or not the second query fingerprint data matches the received second reference fingerprint data to at least one matching threshold value by determining if a sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces at least one maximum value at least as great as the at least one matching threshold value; and
in response to switching from the first frame sampling rate to the second, higher frame sampling rate, and prior to making the second determination, reducing the at least one matching threshold value to increase a tolerance to mis-matching of the second query fingerprint data to the received second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter.
2. The method of claim 1 , wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises computing a sliding window correlation of structured data of a second query fingerprint to structured data of the second reference fingerprint data.
3. The method of claim 1 , wherein the identification of the channel based on the match between the first reference fingerprint data representing a first transmitted portion of the content data transmitted by the content-distribution system and the first query fingerprint data representing at least a portion of the received content data comprises a cold match between first reference fingerprint data and the first query fingerprint data to at least a first threshold level of quantitative agreement,
and wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises reducing the matching threshold value to less than the first threshold level.
4. The method of claim 1 ,
wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises:
determining a temporal alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields a maximum correlation; and
determining whether or not the maximum correlation is at least as large as the matching threshold value.
5. The method of claim 4 , wherein the received content data comprises video data, and wherein content frames in the sequence of content frames comprise video frames.
6. The method of claim 5 , wherein each reference fingerprint in sequence of the second reference fingerprint data comprises a global intensity of a respective video frame of the second transmitted portion of the content data, and a set of local intensities and corresponding spatial frame coordinates of the respective video frame,
wherein each second query fingerprint comprises a global intensity of a respective second query video frame of the second subset of content frames, and the set of local intensities and corresponding spatial frame coordinates of the respective second query video frame,
and wherein determining the alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields the maximum correlation comprises computing a hamming distance between each second query fingerprint and at least one of the second reference fingerprints.
7. The method of claim 1 ,
wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises:
determining that temporal irregularities in the sequence of second query fingerprints cause temporal misalignments between the sequence of second query fingerprints and the sequence of reference fingerprints of the second reference fingerprint data, the temporal misalignments causing mis-matching of at least some second query fingerprints to reference fingerprints;
computing a correlation between the sequence of second query fingerprints and the sequence of reference fingerprints, the correlation including computational effects of the mis-matching second query fingerprints and reference fingerprints; and
determining the second query fingerprint data to be a sufficient match to the second reference fingerprint data if the correlation is at least as great as the reduced matching threshold value.
8. A content-presentation device of a content-modification system that includes a content-distribution system, the content-presentation device comprising:
one or more processors; and
a non-transitory computer-readable storage medium, having stored thereon program instructions that, upon execution by the one or more processors, cause the content-presentation device to carry out operations including:
making a first determination of an upcoming content-modification opportunity in content data received by the content-presentation device on an identified channel from the content-distribution system, wherein identification of the channel is based on a match between first reference fingerprint data representing a first transmitted portion of content data transmitted by the content-distribution system and first query fingerprint data representing at least a portion of the received content data, wherein the received content data comprises a sequence of content frames received at a source frame rate, and wherein the first query fingerprint data comprise a sequence of first query fingerprints that are generated by the content-presentation device from a first sub-sequence of content frames by sampling a first received portion of the sequence of content frames at a first frame sampling rate that is less than the source frame rate;
receiving second reference fingerprint data representing a second transmitted portion of the content data transmitted by the content-distribution system, the second reference fingerprint data comprising a sequence of reference fingerprints received at the source frame rate, wherein the second transmitted portion follows the first transmitted portion;
in response to receiving the second reference fingerprint data: (i) switching from the first frame sampling rate to a second, higher frame sampling rate, wherein the second frame sampling rate is no greater than the source frame rate, (ii) obtaining a second sub-sequence of content frames by sampling a second received portion of the received sequence of content frames at the second frame sampling rate, wherein the second received portion follows the first received portion, (iii) generating second query fingerprint data from the second sub-sequence of content frames, the second query fingerprint data comprising a sequence of second query fingerprints corresponding to the second sub-sequence of content frames, and (iv) making a second determination of whether or not the second query fingerprint data matches the received second reference fingerprint data to at least one matching threshold value by determining if a sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces at least one maximum value at least as great as the at least one matching threshold value; and
in response to switching from the first frame sampling rate to the second, higher frame sampling rate, and prior to making the second determination, reducing the at least one matching threshold value to increase a tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter.
9. The content-presentation device of claim 8 , wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises computing a sliding window correlation of structured data of a second query fingerprint to structured data of the second reference fingerprint data.
10. The content-presentation device of claim 8 , wherein the identification of the channel based on the match between the first reference fingerprint data representing a first transmitted portion of the content data transmitted by the content-distribution system and the first query fingerprint data representing at least a portion of the received content data comprises a cold match between first reference fingerprint data and the first query fingerprint data to at least a first threshold level of quantitative agreement,
and wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises reducing the matching threshold value to less than the first threshold level.
11. The content-presentation device of claim 8 ,
wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises:
determining a temporal alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields a maximum correlation; and
determining whether or not the maximum correlation is at least as large as the matching threshold value.
12. The content-presentation device of claim 11 , wherein the received content data comprises video data, and wherein content frames in the sequence of content frames comprise video frames.
13. The content-presentation device of claim 12 , wherein each reference fingerprint in the sequence of the second reference fingerprint data comprises a global intensity of a respective video frame of the second transmitted portion of the content data, and a set of local intensities and corresponding spatial frame coordinates of the respective video frame,
wherein each second query fingerprint comprises a global intensity of a respective second query video frame of the second subset of content frames, and the set of local intensities and corresponding spatial frame coordinates of the respective second query video frame,
and wherein determining the alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields the maximum correlation comprises computing a hamming distance between each second query fingerprint and at least one of the second reference fingerprints.
14. The content-presentation device of claim 8 ,
wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises:
determining that temporal irregularities in the sequence of second query fingerprints cause temporal misalignments between the sequence of second query fingerprints and the sequence of reference fingerprints of the second reference fingerprint data, the temporal misalignments causing mis-matching of at least some second query fingerprints to reference fingerprints;
computing a correlation between the sequence of second query fingerprints and the sequence of reference fingerprints, the correlation including computational effects of the mis-matching second query fingerprints and reference fingerprints; and
determining the second query fingerprint data to be a sufficient match to the second reference fingerprint data if the correlation is at least as great as the reduced matching threshold value.
15. A non-transitory computer-readable storage medium, having stored thereon program instructions that, upon execution by one or more processors of a content-presentation device of a content-modification system that includes a content-distribution system, cause the content-presentation device to carry out operations including:
making a first determination of an upcoming content-modification opportunity in content data received by the content-presentation device on an identified channel from the content-distribution system, wherein identification of the channel is based on a match between first reference fingerprint data representing a first transmitted portion of the content data transmitted by the content-distribution system and first query fingerprint data representing at least a portion of the received content data, wherein the received content data comprises a sequence of content frames received at a source frame rate, and wherein the first query fingerprint data comprise a sequence of first query fingerprints that are generated by the content-presentation device from a first sub-sequence of content frames by sampling a first received portion of the sequence of content frames at a first frame sampling rate that is less than the source frame rate;
receiving second reference fingerprint data representing a second transmitted portion of the content data transmitted by the content-distribution system, the second reference fingerprint data comprising a sequence of reference fingerprints received at the source frame rate, wherein the second transmitted portion follows the first transmitted portion;
in response to receiving the second reference fingerprint data: (i) switching from the first frame sampling rate to a second, higher frame sampling rate, wherein the second frame sampling rate is no greater than the source frame rate, (ii) obtaining a second sub-sequence of content frames by sampling a second received portion of the received sequence of content frames at the second frame sampling rate, wherein the second received portion follows the first received portion, (iii) generating second query fingerprint data from the second sub-sequence of content frames, the second query fingerprint data comprising a sequence of second query fingerprints corresponding to the second sub-sequence of content frames, and (iv) making a second determination of whether or not the second query fingerprint data matches the received second reference fingerprint data to at least one matching threshold value by determining if a sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces at least one maximum value at least as great as the at least one matching threshold value; and
in response to switching from the first frame sampling rate to the second, higher frame sampling rate, and prior to making the second determination, reducing the at least one matching threshold value to increase a tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter.
16. The non-transitory computer-readable storage medium of claim 15 , wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises computing a sliding window correlation of structured data of a second query fingerprint to structured data of the second reference fingerprint data.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the identification of the channel based on the match between the first reference fingerprint data representing a first transmitted portion of the content data transmitted by the content-distribution system and the first query fingerprint data representing at least a portion of the received content data comprises a cold match between first reference fingerprint data and the first query fingerprint data to at least a first threshold level of quantitative agreement,
and wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises reducing the matching threshold value to less than the first threshold level.
18. The non-transitory computer-readable storage medium of claim 15 ,
wherein determining if the sliding window correlation of the sequence of second query fingerprints computed across the sequence of reference fingerprints produces the at least one maximum value at least as great as the at least one matching threshold value comprises:
determining a temporal alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields a maximum correlation; and
determining whether or not the maximum correlation is at least as large as the matching threshold value.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the received content data comprises video data, and wherein content frames in the sequence of content frames comprise video frames,
wherein each reference fingerprint in sequence of the second reference fingerprint data comprises a global intensity of a respective video frame of the second transmitted portion of the content data, and a set of local intensities and corresponding spatial frame coordinates of the respective video frame,
wherein each second query fingerprint comprises a global intensity of a respective second query video frame of the second subset of content frames, and the set of local intensities and corresponding spatial frame coordinates of the respective second query video frame,
and wherein determining the alignment of the sequence of second query fingerprints with the sequence of second reference fingerprints that yields the maximum correlation comprises computing a hamming distance between each second query fingerprint and at least one of the second reference fingerprints.
20. The non-transitory computer-readable storage medium of claim 15 ,
wherein reducing the matching threshold value to increase the tolerance to mis-matching of the second query fingerprint data to the second reference fingerprint data when the second sub-sequence of content frames is subject to frame jitter comprises:
determining that temporal irregularities in the sequence of second query fingerprints cause temporal misalignments between the sequence of second query fingerprints and the sequence of reference fingerprints of the second reference fingerprint data, the temporal misalignments causing mis-matching of at least some second query fingerprints to reference fingerprints;
computing a correlation between the sequence of second query fingerprints and the sequence of reference fingerprints, the correlation including computational effects of the mis-matching second query fingerprints and reference fingerprints; and
determining the second query fingerprint data to be a sufficient match to the second reference fingerprint data if the correlation is at least as great as the reduced matching threshold value.