Website classification via containment queries
There is disclosed a computer-implemented system and method for categorizing unclassified websites, including computing a data sketch of an unclassified website, wherein the data sketch is a probabilistic data structure that represents text of the unclassified website; computing an intersection value between the data sketch and a comparison set, wherein the comparison set comprises data sketches of known, classified websites; assigning matches for a subset of websites from the comparison set, wherein the subset includes intersections above an intersection threshold; classifying the website into a category of a website from the subset; and performing web traffic management based on the classification.
1 . A computer-implemented method of categorizing unclassified websites, comprising:
computing a data sketch of an unclassified website, wherein the data sketch comprises a probabilistic data structure that represents text of the unclassified website;
computing an intersection value between the data sketch and a comparison set, wherein the comparison set comprises data sketches of known, classified websites, wherein computing the intersection comprises computing a containment query wherein the containment query is computed according to an intersection of the data sketch and known, categorized websites in the comparison set, divided by a size of the comparison set;
assigning matches for a subset of websites from the comparison set, wherein the subset includes intersections above an intersection threshold;
classifying the website into a category of a website from the subset; and
performing web traffic management based on the classification.
2 . The method of claim 1 , wherein the data sketch is computed according to a MinHash.
3 . The method of claim 1 , wherein computing the intersection comprises performing locality sensitive hashing.
4 . The method of claim 1 , further comprising assigning matches for a plurality of subsets, wherein a first subset represents a first intersection threshold, a second subset represents a second intersection threshold, and a third subset represents a third intersection threshold.
5 . The method of claim 4 , wherein the first intersection threshold is substantially 50%.
6 . The method of claim 4 , wherein the second intersection threshold is substantially 30%.
7 . The method of claim 4 , wherein the third intersection threshold is substantially 10%.
8 . The method of claim 4 , further comprising selecting only the subset with a highest intersection threshold.
9 . The method of claim 1 , further comprising providing a multinominal classifier, and assigning to the unclassified website multiple categories selected from the subset.
10 . The method of claim 1 , further comprising providing a unitary classifier, and assigning to the unclassified website a single category selected from the subset.
11 . The method of claim 1 , wherein the text is plain text.
12 . The method of claim 1 , further comprising selecting the unclassified website for analysis based on crowd-sourcing unknown websites visited by users.
13 . The method of claim 1 , wherein the comparison set includes at least some human-assigned classifications.
14 . One or more tangible, nontransitory computer-readable storage media having stored thereon executable instructions to:
compute a data sketch of an unclassified website, wherein the data sketch comprises a probabilistic data structure that represents text of the unclassified website;
compute an intersection value between the data sketch and a comparison set, wherein the comparison set comprises data sketches of known, classified websites, and wherein computing the intersection comprises computing a containment query, wherein the containment query is computed according to an intersection of the data sketch and known, categorized websites in the comparison set, divided by a size of the comparison set;
assign matches for a subset of websites from the comparison set, wherein the subset includes intersections above an intersection threshold;
classify the website into a category of a website from the subset; and
perform web traffic management based on the classifying.
15 . The one or more tangible, nontransitory computer-readable storage media of claim 14 , wherein the data sketch is computed according to a MinHash.
16 . The one or more tangible, nontransitory computer-readable storage media of claim 14 , wherein computing the intersection comprises performing locality sensitive hashing.
17 . A computing apparatus, comprising:
a hardware platform comprising a processor circuit and a memory; and
instructions encoded within the memory to instruct the processor circuit to:
compute a data sketch of an unclassified website, wherein the data sketch comprises a probabilistic data structure that represents text of the unclassified website;
compute an intersection value between the data sketch and a comparison set, wherein the comparison set comprises data sketches of known, classified websites, and wherein computing the intersection comprises computing a containment query, wherein the containment query is computed according to an intersection of the data sketch and known, categorized websites in the comparison set, divided by a size of the comparison set;
assign matches for a subset of websites from the comparison set, wherein the subset includes intersections above an intersection threshold;
classify the website into a category of a website from the subset; and
perform web traffic management based on the classifying.
18 . The computing apparatus of claim 17 , wherein the data sketch is computed according to a MinHash.
19 . The computing apparatus of claim 17 , wherein the instructions are further to assign matches for a plurality of subsets, wherein a first subset represents a first intersection threshold, a second subset represents a second intersection threshold, and a third subset represents a third intersection threshold.
20 . The computing apparatus of claim 19 , wherein the first intersection threshold is substantially 50%.