Leveraging audio matches to improve visual matching recall between video content items

1 . A method, comprising:

performing an audio matching between a first video content item and a second video content item to identify a matching audio segment;

determining first temporal boundaries within the first video content item and second temporal boundaries within the second video content item corresponding to the identified matching audio segment;

performing a visual matching between the first video content item within the first temporal boundaries and the second video content item within the second temporal boundaries using a modified visual similarity threshold that is lower than a baseline visual similarity threshold; and

determining whether a match exists between the first and second video content items based on the visual matching.

2 . The method of claim 1 , wherein performing the visual matching comprises:

analyzing columns and rows of a visual similarity heatmap at timepoints corresponding to the matching audio segment; and

determining whether high similarity values are concentrated along a diagonal band within the columns and the rows relative to surrounding areas of the visual similarity heatmap.

3 . The method of claim 1 , wherein performing the visual matching comprises:

generating visual fingerprints for portions of the first video content item and the second video content item within respective first temporal boundaries and the second temporal boundaries; and

computing visual similarities between the visual fingerprints.

4 . The method of claim 1 , wherein the matching audio segment comprises multiple non-contiguous audio segments within the first and second video content items.

5 . The method of claim 3 , wherein performing the visual matching comprises analyzing a distribution of the visual similarities relative to temporal alignment between the first and second video content items.

6 . The method of claim 3 , wherein determining whether the match exists comprises detecting a concentration of the visual similarities along a temporal progression corresponding to the matching audio segment.

7 . The method of claim 5 , wherein analyzing the distribution comprises: identifying

a top predetermined percentage of highest visual similarity values; and

determining whether the top predetermined percentage of similarity values are concentrated along a temporal alignment corresponding to the matching audio segment.

8 . A system, comprising:

one or more memories; and

one or more processors, the one or more processors configured to execute instructions stored in the one or more memories to:

perform an audio matching between a first video content item and a second video content item to identify a matching audio segment;

determine first temporal boundaries within the first video content item and second temporal boundaries within the second video content item corresponding to the identified matching audio segment;

perform a visual matching between the first video content item within the first temporal boundaries and the second video content item within the second temporal boundaries using a modified visual similarity threshold that is lower than a baseline visual similarity threshold; and

determine whether a match exists between the first and second video content items based on the visual matching.

9 . The system of claim 8 , wherein to perform the visual matching comprises instructions to:

analyze columns and rows of a visual similarity heatmap at timepoints corresponding to the matching audio segment; and

determine whether high similarity values are concentrated along a diagonal band within the columns and the rows relative to surrounding areas of the visual similarity heatmap.

10 . The system of claim 8 , wherein to perform the visual matching comprises instructions to:

generate visual fingerprints for portions of the first video content item and the second video content item within respective first temporal boundaries and the second temporal boundaries; and

compute visual similarities between the visual fingerprints.

11 . The system of claim 8 , wherein to determine the first temporal boundaries and the second temporal boundaries comprises instructions to:

identify start and end times of the matching audio segment within each of the first and second video content items; and

extend the temporal boundaries beyond the start and end times by a predetermined time interval.

12 . The system of claim 8 , wherein to determine whether the match exists comprises instructions to:

identify a percentage of visual similarities that exceed the modified visual similarity threshold; and

compare the percentage to a threshold percentage.

13 . The system of claim 8 , wherein the first video content item comprises a query video being checked for unauthorized use of content from the second video content item.

14 . The system of claim 10 , wherein to perform the visual matching comprises instructions to:

analyze a distribution of the visual similarities relative to temporal alignment between the first and second video content items.

15 . The system of claim 10 , wherein to determine whether the match exists comprises instructions to:

detect a concentration of the visual similarities along a temporal progression corresponding to the matching audio segment.

16 . The system of claim 14 , wherein to analyze the distribution comprises instructions to:

identify a top predetermined percentage of highest visual similarity values; and

determine whether the top predetermined percentage of similarity values are concentrated along a temporal alignment corresponding to the matching audio segment.

17 . One or more non-transitory computer readable media storing instructions operable to cause one or more processors to perform operations comprising:

performing an audio matching between a first video content item and a second video content item to identify a matching audio segment;

determining first temporal boundaries within the first video content item and second temporal boundaries within the second video content item corresponding to the identified matching audio segment;

performing a visual matching between the first video content item within the first temporal boundaries and the second video content item within the second temporal boundaries using a modified visual similarity threshold that is lower than a baseline visual similarity threshold; and

determining whether a match exists between the first and second video content items based on the visual matching.

18 . The one or more non-transitory computer readable media of claim 17 , wherein performing the visual matching comprises:

analyzing columns and rows of a visual similarity heatmap at timepoints corresponding to the matching audio segment; and

determining whether high similarity values are concentrated along a diagonal band within the columns and the rows relative to surrounding areas of the visual similarity heatmap.

19 . The one or more non-transitory computer readable media of claim 17 , wherein performing the visual matching comprises:

generating visual fingerprints for portions of the first video content item and the second video content item within respective first temporal boundaries and the second temporal boundaries; and

computing visual similarities between the visual fingerprints.

20 . The one or more non-transitory computer readable media of claim 19 , wherein performing the visual matching comprises analyzing a distribution of the visual similarities relative to temporal alignment between the first and second video content items.