Software web crowler and method therefor
View Patent ↗System for crawling the web for multimedia files and indexing the files based on sound analysis and algorithmic translation.
1. A computerized system comprising:
a crawler obtaining multimedia files from a network, the crawler comprising:
a multi-threaded downloader that downloads web pages;
a queue storing links corresponding to the download web pages;
a scheduler obtaining the stored links from the queue and passing the obtained links to the multi-threaded downloader, wherein the multi-threaded downloader downloads multiple multimedia files concurrently from said links;
a multimedia processor receiving said multimedia files from the crawler and processing said multimedia files by translating speech in the multimedia files into a textual representation,
wherein said multimedia processor determines sound effects in said multimedia files by comparing said sound effects in said multimedia files against a predetermined set of sounds, wherein generated metadata is determined by the comparison, and wherein said metadata comprises keywords identifying a type of said sound effects;
a data mining module that extracts text information from the textual representation; and
an indexer that indexes the multimedia files based on said keywords and said text information.
2. The computerized system of claim 1 , wherein the multimedia processor is further operable to detect a type of music in the multimedia files.
3. The computerized system of claim 1 , wherein the data mining algorithm:
a. detects frequency of keywords in the generated metadata;
b. adds weights to the keywords, and
c. compiles the keywords for each multimedia file.
4. The computerized system of claim 1 , wherein the multimedia files are audio files.
5. The computerized system of claim 1 , wherein the multimedia files are video files.
6. A computer implemented method comprising:
obtaining multimedia files from a network by utilizing a multi-threaded downloader that download multiple multimedia files concurrently from links acquired from downloaded web pages;
processing, using a processor, the obtained multimedia files by translating speech in the multimedia files into a textual representation,
wherein said processing further determines sound effects in said multimedia files by comparing said sound effects in said multimedia files against a predetermined set of sounds, wherein generated metadata is determined by the comparison, and wherein said metadata comprises keywords identifying a type of said sound effects;
extracting text information from the textual representation, and;
indexing the multimedia files based on said keywords and said text information.
7. The method of claim 6 , wherein processing further comprises detecting a type of music in the multimedia files.
8. The method of claim 6 , wherein extracting further comprises:
a. detecting frequency of keywords in the generated metadata;
b. adding weights to the keywords; and
c. compiling the keywords for each multimedia file.
9. The computerized system of claim 1 , wherein the multimedia files are audio files.
10. The computerized system of claim 1 , wherein the multimedia files are video files.