Spam detection for user-generated multimedia items based on concept clustering
A system, a method, and various software tools enable a video hosting website to automatically identify posted video items that contain spam in the metadata associated with a respective video item. A spam detection tool for user-generated video items based on concept clustering is provided that facilitates the detection of spam in the metadata associated with a video item.
1. A computer-implemented method for processing a multimedia item, comprising:
storing the multimedia item and associated metadata in a multimedia database, wherein the metadata includes a plurality of tokens;
determining, for each of the plurality of tokens, a concept associated with the token;
determining a number of related concepts from concepts determined from the plurality of tokens included in the metadata;
clustering the related concepts into corresponding clusters; and
responsive to a number of distinct clusters exceeding a threshold, marking the multimedia item as spam.
2. The computer-implemented method of claim 1 , wherein the multimedia item is a video.
3. The computer-implemented method of claim 1 , further comprising:
removing the multimedia item from the multimedia database when the multimedia item is marked as spam.
4. The computer-implemented method of claim 1 , further comprising:
refraining from indexing the multimedia item if the multimedia item is marked as spam.
5. The computer-implemented method of claim 1 , wherein the metadata is provided by a user who uploaded the multimedia item to a hosting website.
6. The computer-implemented method of claim 1 , wherein the threshold is based on a distribution of the number of related concepts from the item description that are not indicative of spam.
7. The computer-implemented method of claim 1 , wherein the threshold is based on a mean number of unrelated concepts.
8. A computer-implemented method for processing a multimedia item, comprising:
storing the multimedia item and associated metadata in a multimedia database, wherein the metadata includes a plurality of tokens;
determining, for each of the plurality of tokens, a concept associated with the token;
determining a number of related concepts from concepts determined from the plurality of tokens included in the metadata;
clustering the related concepts into corresponding clusters; and
responsive to determining at least one combination of distinct clusters, marking the multimedia item as spam.
9. The computer-implemented method of claim 8 , wherein the multimedia item is a video.
10. The computer-implemented method of claim 8 , further comprising:
removing the multimedia item from the multimedia database when the multimedia item is marked as spam.
11. The computer-implemented method of claim 8 , further comprising:
refraining from indexing the multimedia item if the multimedia item is marked as spam.
12. A system for processing a multimedia item, the system comprising:
a computer processor; and
a computer-readable storage medium storing executable code, the code when executed by the processor performs steps comprising:
storing the multimedia item and associated metadata in a multimedia database, wherein the metadata includes a plurality of tokens;
determining, for each of the plurality of tokens, a concept associated with the token;
determining a number of related concepts concepts from concepts determined from the plurality of tokens included in the metadata;
clustering the related concepts into corresponding clusters; and
marking the multimedia item as spam responsive to a number of distinct clusters exceeding a threshold.
13. The system of claim 12 , wherein the multimedia item is a video.
14. The system of claim 12 , wherein the code when executed by the processor performs further steps comprising:
removing the multimedia item from the multimedia database when the multimedia item is marked as spam.
15. The system of claim 12 , wherein the code when executed by the processor performs further steps comprising:
refraining from indexing the multimedia item when the multimedia item is marked as spam.
16. A non-transitory computer-readable storage medium containing program code for processing a multimedia item, the program code when executed performs steps comprising:
storing the multimedia item and associated metadata in a multimedia database, wherein the metadata includes a plurality of tokens;
determining, for each of the plurality of tokens, a concept associated with the token;
determining a number of related concepts from concepts determined from the plurality of tokens included in the metadata;
clustering the related concepts into corresponding clusters; and
responsive to a number of distinct clusters exceeding a threshold, marking the multimedia item as spam.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the multimedia item is a video.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the program code when executed performs further steps comprising:
removing the multimedia item from the multimedia database when the multimedia item is marked as spam.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the program code when executed performs further steps comprising:
refraining from indexing the multimedia item when the multimedia item is marked as spam.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the threshold is based on a mean number of unrelated concepts.