Focused URL recrawl
A website misclassification report is received. A determination is made that a current classification model correctly classifies a website. The current classification model is different from a model that was previously used to classify the website. In response to a determination that the website is correctly classified by the current classification model, a reclassification operation is performed, using the current classification model, on a second website.
1. A system, comprising:
a processor configured to:
in response to receiving a website misclassification report comprising a user-reported indication that access to a first URL, having an associated first domain, is erroneously blocked, wherein the first URL was previously assigned a categorization associated with at least a first subject matter topic based on content analysis performed by an original classification model, use a single page classifier to perform a recrawl-reclassification operation on the first URL using a current classification model that is an updated version of the original classification model, to determine at least one current subject matter topic and assign a current categorization for the first URL; and
determine that there is at least one discrepancy among at least two of: the first categorization, the current categorization, or information included in the misclassification report, and take a remedial action in response to the determination, wherein the remedial action includes at least one of: (1) initiating an escalation event when the first categorization and the result of the recrawl-reclassification operation are in agreement, or (2) assigning the current categorization to the URL when the current categorization is different from the first categorization and initiating a recrawl-reclassification operation on at least one additional domain; and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system of claim 1 , wherein the website misclassification report is received in response to the user interacting with a security appliance.
3. The system of claim 1 , wherein the website misclassification report is received at least in part as a URL input, by the user, into an interactive form served by the system.
4. The system of claim 1 , wherein the website misclassification report includes a second categorization as a suggested correct classification for the first URL.
5. The system of claim 1 , wherein the determination of whether the current classification model correctly classifies the first URL includes determining whether the current classification model's classification matches a suggested classification provided with the website misclassification report.
6. The system of claim 1 , wherein the system is further configured to classify a corpus of documents and further configured to generate an index comprising a subset of documents included in the corpus.
7. The system of claim 6 , wherein the corpus includes websites classified as malicious and wherein the index does not include websites classified as malicious.
8. The system of claim 6 , wherein the corpus includes websites classified as phishing and wherein the index does not include websites classified as phishing.
9. The system of claim 6 , wherein the corpus includes websites that result in a soft 404 error message when an access attempt is made and wherein the index does not include websites that result in a soft 404 error message when an access attempt is made.
10. A method, comprising:
in response to receiving a website misclassification report comprising a user-reported indication that access to a first URL, having an associated first domain, is erroneously blocked, wherein the first URL was previously assigned a categorization associated with at least a first subject matter topic based on content analysis performed by an original classification model, using a single page classifier to perform a recrawl-reclassification operation on the first URL using a current classification model that is an updated version of the original classification model, to determine at least one current subject matter topic and assign a current categorization for the first URL; and
determining that there is at least one discrepancy among at least two of: the first categorization, the current categorization, or information included in the misclassification report, and taking a remedial action in response to the determination, wherein the remedial action includes at least one of: (1) initiating an escalation event when the first categorization and the result of the recrawl-reclassification operation are in agreement, or (2) assigning the current categorization to the URL when the current categorization is different from the first categorization and initiating a recrawl-reclassification operation on at least one additional domain.
11. The method of claim 10 , wherein the website misclassification report is received in response to the user interaction with a security appliance.
12. The method of claim 10 , wherein the website misclassification report is received at least in part as a URL input, by the user, into an interactive form served by the system.
13. The method of claim 10 , wherein the website misclassification report includes a second categorization as a suggested correct classification for the first URL.
14. The method of claim 10 , wherein the determination of whether the current classification model correctly classifies the first URL includes determining whether the current classification model's classification matches a suggested classification provided with the website misclassification report.
15. The method of claim 10 , further comprising classifying a corpus of documents and generating an index comprising a subset of documents included in the corpus.
16. The method of claim 15 , wherein the corpus includes websites classified as malicious and wherein the index does not include websites classified as malicious.
17. The method of claim 15 , wherein the corpus includes websites classified as phishing and wherein the index does not include websites classified as phishing.
18. The method of claim 15 , wherein the corpus includes websites that result in a soft 404 error message when an access attempt is made and wherein the index does not include websites that result in a soft 404 error message when an access attempt is made.
19. A computer program product embodied in a non-transitory tangible computer readable storage medium and comprising computer instructions for:
in response to receiving a website misclassification report comprising a user-reported indication that access to a first URL, having an associated first domain, is erroneously blocked, wherein the first URL was previously assigned a categorization associated with at least a first subject matter topic based on content analysis performed by an original classification model, using a single page classifier to perform a recrawl-reclassification operation on the first URL using a current classification model that is an updated version of the original classification model, to determine at least one current subject matter topic and assign a current categorization for the first URL; and
determining that there is at least one discrepancy among at least two of: the first categorization, the current categorization, or information included in the misclassification report, and taking a remedial action in response to the determination, wherein the remedial action includes at least one of: (1) initiating an escalation event when the first categorization and the result of the recrawl-reclassification operation are in agreement, or (2) assigning the current categorization to the URL when the current categorization is different from the first categorization and initiating a recrawl-reclassification operation on at least one additional domain.