Methods for detecting and removing duplicates in video search results
View Patent ↗Duplicate video search results are detected and removed. Digital signatures are generated for each video content item of a video content corpus. Duplicates are determined for the top n previously received queries by determining the similarity of video content items that are within the same results set of each particular query of the top n previously received queries. Similarities are calculated between any two video documents of the result set of the particular query by measuring the difference between the digital signatures of two video documents. If a similarity between two videos is determined to be above a particular threshold, then the two videos are considered duplicates of each other and the search index is updated by retaining the most relevant of the video documents to the particular query. The less relevant video documents are flagged as duplicates with respect to the particular query.
1. A computer-implemented method for removing duplicate content items in a result set, the method comprising:
identifying a subset of content items of a search index based at least upon content items returned in a result set generated by a search engine for a particular query that was previously submitted to the search engine;
for each content item in the subset, determining a similarity value with respect to all other content items in the subset;
upon determining that a similarity value of a first content item and a second content item of the subset of content items satisfy a particular threshold value, determining which of the first content item and second content item are less relevant to the particular query; and
marking, in the search index, the less relevant to the particular query of the first content item and second content item as a duplicate content item for the particular query;
in response to determining that the more relevant of the first content item and the second content item to the particular query is no longer available:
determining which of the content items are marked as a duplicate and satisfy the particular threshold value is the most relevant content item to the particular query; and
updating the search index by removing the mark as duplicate for the most relevant content item to the particular query;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the content items are multimedia video content items.
3. The method of claim 1 , wherein determining a similarity value further comprises determining a similarity value for the top x content items in the subset based upon relevance.
4. The method of claim 1 , wherein the method is performed on only the most popular n queries previously received.
5. The method of claim 1 , wherein determining a similarity value further comprises performing:
generating a digital signature for each content item; and
calculating the difference in bits between the digital signature for one content item to the digital signature of another content item.
6. The method of claim 1 , wherein the similarity value is the hamming distance between content items.
7. The method of claim 1 , further comprising:
removing, from the search index, the more relevant of the first content item and second content item to the particular query that is no longer available.
8. The method of claim 1 , wherein the method is performed offline prior to receiving a query from a user.
9. One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause one or more computing devices to perform:
identifying a subset of content items of a search index based at least upon content items returned in a result set generated by a search engine for a particular query that was previously submitted to the search engine;
for each content item in the subset, determining a similarity value with respect to all other content items in the subset;
upon determining that a similarity value of a first content item and a second content item of the subset of content items satisfy a particular threshold value,
determining which of the first content item and second content item are less relevant to the particular query; and
marking, in the search index, the less relevant to the particular query of the first content item and second content item as a duplicate content item for the particular query;
in response to determining that the more relevant of the first content item and the second content item to the particular query is no longer available:
determining which of the content items are marked as a duplicate and satisfy the particular threshold value is the most relevant content item to the particular query; and
updating the search index by removing the mark as duplicate for the most relevant content item to the particular query.
10. The one or more non-transitory computer-readable media of claim 9 , wherein the content items are multimedia video content items.
11. The one or more non-transitory computer-readable media of claim 9 , wherein instructions for determining a similarity value further comprises instructions for determining a similarity value for the top x content items in the subset based upon relevance.
12. The one or more non-transitory computer-readable media of claim 9 , wherein the instructions cause said selecting to be performed on the most popular n queries previously received.
13. The one or more non-transitory computer-readable media of claim 9 , wherein instructions for determining a similarity value further comprises instructions for performing:
generating a digital signature for each content item; and
calculating the difference in bits between the digital signature for one content item to the digital signature of another content item.
14. The one or more non-transitory computer-readable media of claim 9 , wherein the similarity value is the hamming distance between content items.
15. The one or more non-transitory computer-readable media of claim 9 , wherein the instructions, when executed by one or more processors, further cause the one or more computing devices to perform:
removing, from the search index, the more relevant of the first content item and second content item to the particular query that is no longer available.
16. The one or more non-transitory computer-readable media of claim 9 , wherein the method is performed offline prior to receiving a query from a user.