SYSTEMS AND METHODS FOR EXTRAPOLATING FROM CRAWLED DATA TO GENERATE CLASSIFICATIONS
Disclosed herein are systems and method for extrapolating from crawled data to generate classifications. In one aspect, a method may receive an evaluation request with an input list comprising at least one entity and a respective website identifier of the at least one entity, identify, from a plurality of categorization labels, a subset of categorization labels that correspond to a requesting entity that generated the evaluation request, crawl information using the website identifier, generate at least one text body from the crawled information, apply a machine learning algorithm on the at least one text body, wherein the machine learning algorithm is configured to generate an output vector indicating categorization labels from the subset of categorization labels that the at least one entity corresponds to, based on terms in the at least one text body; and transmit the output vector to a computing device of the requesting entity.
1 . A method for extrapolating from crawled data to generate classifications, the method comprising:
receiving an evaluation request with an input list comprising at least one entity and a respective website identifier of the at least one entity;
identifying, from a plurality of categorization labels, a subset of categorization labels that correspond to a requesting entity that generated the evaluation request;
crawling information from at least one website corresponding to the website identifier;
generating at least one text body by parsing the crawled information;
applying a machine learning algorithm on the at least one text body, wherein the machine learning algorithm is configured to generate an output vector indicating categorization labels from the subset of categorization labels that the at least one entity corresponds to, based on terms in the at least one text body; and
transmitting the output vector to a computing device of the requesting entity.
2 . The method of claim 1 , wherein the machine learning algorithm is further configured to:
weight each term in the at least one text body based on whether the term is present in a list of critical terms for the subset of categorization labels; and
determine, based on the weighted terms, whether the at least one text body corresponds to a given categorization label.
3 . The method of claim 1 , wherein the machine learning algorithm is further configured to:
calculate a ratio of an amount of critical terms in the at least one text body and a total amount of terms in the at least one text body and; and
determine whether the at least one text body corresponds to a given categorization label further based on the ratio.
4 . The method of claim 1 , wherein parsing the crawled information further comprises:
removing stop words and punctuation from the crawled information; and
classifying, using object recognition, images from the at least one website into text describing contents of the images.
5 . The method of claim 1 , wherein the evaluation request comprises an evaluation type, and wherein the subset of categorization labels are further identified based on a compatibility with the evaluation type.
6 . The method of claim 5 , wherein the evaluation type is one of a partner and a client.
7 . The method of claim 1 , wherein a different subset of categorization labels are used for a different requesting entity.
8 . The method of claim 1 , wherein crawling the information further comprises utilizing a proxy service that hides an IP address of a web crawler.
9 . The method of claim 1 , wherein crawling the information further comprises executing a script that circumvents security measures of the at least one website.
10 . A system for extrapolating from crawled data to generate classifications, the system comprising:
at least one memory; and
at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to:
receive an evaluation request with an input list comprising at least one entity and a respective website identifier of the at least one entity;
identify, from a plurality of categorization labels, a subset of categorization labels that correspond to a requesting entity that generated the evaluation request;
crawl information from at least one website corresponding to the website identifier;
generate at least one text body by parsing the crawled information;
apply a machine learning algorithm on the at least one text body, wherein the machine learning algorithm is configured to generate an output vector indicating categorization labels from the subset of categorization labels that the at least one entity corresponds to, based on terms in the at least one text body; and
transmit the output vector to a computing device of the requesting entity.
11 . The system of claim 10 , wherein the machine learning algorithm is further configured to:
weight each term in the at least one text body based on whether the term is present in a list of critical terms for the subset of categorization labels; and
determine, based on the weighted terms, whether the at least one text body corresponds to a given categorization label.
12 . The system of claim 10 , wherein the machine learning algorithm is further configured to:
calculate a ratio of an amount of critical terms in the at least one text body and a total amount of terms in the at least one text body and; and
determine whether the at least one text body corresponds to a given categorization label further based on the ratio.
13 . The system of claim 10 , wherein the at least one hardware processor is configured to parse the crawled information by:
removing stop words and punctuation from the crawled information; and
classifying, using object recognition, images from the at least one website into text describing contents of the images.
14 . The system of claim 10 , wherein the evaluation request comprises an evaluation type, and wherein the subset of categorization labels are further identified based on a compatibility with the evaluation type.
15 . The system of claim 14 , wherein the evaluation type is one of a partner and a client.
16 . The system of claim 10 , wherein a different subset of categorization labels are used for a different requesting entity.
17 . The system of claim 10 , wherein the at least one hardware processor is configured to crawl the information by utilizing a proxy service that hides an IP address of a web crawler.
18 . The system of claim 10 , wherein the at least one hardware processor is configured to crawl the information by executing a script that circumvents security measures of the at least one website.
19 . A non-transitory computer readable medium storing thereon computer executable instructions for extrapolating from crawled data to generate classifications, including instructions for:
receiving an evaluation request with an input list comprising at least one entity and a respective website identifier of the at least one entity;
identifying, from a plurality of categorization labels, a subset of categorization labels that correspond to a requesting entity that generated the evaluation request;
crawling information from at least one website corresponding to the website identifier;
generating at least one text body by parsing the crawled information;
applying a machine learning algorithm on the at least one text body, wherein the machine learning algorithm is configured to generate an output vector indicating categorization labels from the subset of categorization labels that the at least one entity corresponds to, based on terms in the at least one text body; and
transmitting the output vector to a computing device of the requesting entity.