Advanced data collection block identification
Systems and methods that allow examination of response data collected from content providers and provide for classification and routing according to the classification. The process of classification employs an unsupervised, or partially unsupervised, Machine Learning classifier model for identifying data collection responses that contains no data, mangled data, or a block, for assigning a classification correspondingly and for feeding the classification decision back to a data collection platform.
1. A system for classifying data employing a machine learning classification model including a non-transitory computer-readable medium comprising instructions that, when executed by a processor, instruct the processor to operate the system, the system comprising:
at least one service provider infrastructure comprising:
a block detection unit, operable to perform at least:
to label an initial set of data with either a ‘block’ label or a ‘non-block’ label;
upon labeling, to subject the initial set of data to a pre-processing procedure;
to classify the new data by employing the block detection model and to submit a result of classification to a scraping session;
to subject an adaptable percentage of the result of classification to an augmentation process and to integrate the adaptable percentage of the result of classification with a training dataset;
a scraping agent, operable to perform at least one of the following:
to execute the scraping session against a target in response to a scraping request received from a client device;
to receive the result of classification from the block detection unit.
2. The system of claim 1 , wherein the initial set of data is a collection of HyperText Markup Language (HTML) documents aggregated during multiple scraping sessions.
3. The system of claim 1 , wherein the ‘block’ label indicates that the initial set of data comprises data blocked by the target.
4. The system of claim 1 , wherein the ‘non-block’ label indicates that the initial set of data comprises data not blocked by the target.
5. The system of claim 1 , wherein the block detection unit executes the pre-processing procedure by executing at least:
parsing textual elements of the initial set of data;
detecting a language of the textual elements;
modifying the textual elements;
tokenizing the textual elements;
eliminating a first portion of the textual elements that are deemed irrelevant and reducing a second portion of the textual elements to root words; and
translating the textual elements into at least one other language.
6. The system of claim 5 , wherein the textual elements are translated into more languages than the at least one other language.
7. The system of claim 1 , wherein the block detection unit produces the training data set after pre-processing the initial set of data.
8. The system of claim 1 , wherein the block detection model is based on the machine learning classification model.
9. The system of claim 8 , wherein the machine learning classification model may comprise at least one or a combination of the following:
bag of words;
naïve bayes algorithm;
support vector machines;
logistic regression;
random forest classifier;
xtreme gradient boosting model;
convolutional neural network; or
recurrent neural network.
10. The system of claim 1 , wherein the scraping agent submits the new data to the block detection unit for classification after the scraping session.
11. The system of claim 10 , wherein the scraping agent receives the new data from the target as a response to the scraping request submitted by the scraping agent to the target as part of the scraping session.
12. The system of claim 1 , wherein the new data is an HTML document received from the target.
13. The system of claim 1 , wherein the result of classification is a ‘block content’ or a ‘non-block content’.
14. The system of claim 13 , wherein the ‘block content’ implies that the new data comprises data blocked by the target.
15. The system of claim 13 , wherein the ‘non-block content’ implies that the new data comprises data not blocked by the target and suitable for delivering to the client device.
16. The system of claim 1 , wherein the scraping agent delivers the new data to the client device when the result of classification received from the block detection unit is the ‘non-block content’.
17. The system of claim 1 , wherein the scraping agent executes the scraping session on behalf of the client device.
18. The system of claim 1 , wherein the scraping agent analyzes the scraping request and selects a scraping strategy for executing the scraping session.
19. The system of claim 18 , wherein the scraping strategy comprises at least one of:
choosing a scraping agent application;
selecting a proxy server suitable for the scraping request.