Locality-sensitive hashing to clean and normalize text logs
Techniques for improved text normalization are provided. Signatures are generated for a first word and a second word using a locality-sensitive hashing technique. A graph is constructed based on the first and second signatures, by creating a first node in the graph for the first word, creating a second node in the graph for the second word, and creating an edge in the graph connecting the first and second nodes upon determining that the first and second signatures match. A mapping from the first word to the second word is then generated based on the graph.
1. A computer-implemented method, comprising:
receiving an input dataset comprising a plurality of words including word variants of one or more canonical words;
generating a first plurality of signatures for a plurality of words using one or more locality-sensitive hashing techniques including at least an element of randomness;
generating a second plurality of signatures for the plurality of words using the one or more locality-sensitive hashing techniques;
constructing a graph based on the first and second pluralities of signatures, comprising:
creating a respective node in the graph for each respective word from the plurality of words; and
for each respective pair of words with matching signatures in the first or second plurality of signatures, creating a respective edge in the graph;
selecting, for each respective subgraph in the graph, a respective pivot word;
generating, based on the graph, a mapping between the plurality of words and the selected pivot words; and
cleaning the input dataset based on the mapping by replacing the word variants with corresponding pivot words.
2. The computer-implemented method of claim 1 , wherein constructing the graph further comprises:
assigning a respective weight to each respective edge in the graph based on a number of times the corresponding pair of words were assigned matching signatures by the one or more locality-sensitive hashing techniques.
3. The computer-implemented method of claim 2 , wherein constructing the graph further comprises:
identifying one or more edges in the graph with a weight below a predefined pruning criterion; and
pruning the identified one or more edges.
4. The computer-implemented method of claim 1 , the method further comprising:
receiving a second input dataset comprising a second plurality of words;
determining that a first word of the second plurality of words is not included in the graph;
generating a first signature for the first word using the one or more locality-sensitive hashing techniques; and
mapping the first word to a first pivot word based on the first signature.
5. The computer-implemented method of claim 1 , wherein generating the first plurality of signatures using one or more locality-sensitive hashing techniques comprises, for a first word of the plurality of words:
generating a set of sub strings contained within the first word;
generating a plurality of hash values for the set of sub strings;
partitioning the plurality of hash values into a plurality of bins;
selecting a representative value for each respective bin of the plurality of bins, wherein the representative value is a smallest value in the respective bin;
filling one or more empty bins of the plurality of bins based on representative values for adjacent bins; and
generating a first signature for the first word based on hashing the representative values.
6. A non-transitory computer readable medium comprising computer readable code executable by one or more processors to perform an operation comprising:
generating a first signature for a first word and a second signature for a second word using one or more locality-sensitive hashing techniques;
constructing a graph based on the first and second signatures, comprising:
creating a first node in the graph for the first word;
creating a second node in the graph for the second word; and
upon determining that the first and second signatures match, creating an edge in the graph connecting the first and second nodes; and
generating, based on the graph, a mapping from the first word to the second word.
7. The non-transitory computer readable medium of claim 6 , the operation further comprising:
generating a first plurality of signatures for the first word using the one or more locality-sensitive hashing techniques a plurality of times;
generating a second plurality of signatures for the second word using the one or more locality-sensitive hashing techniques a plurality of times; and
updating the graph based on the first and second pluralities of signatures.
8. The non-transitory computer readable medium of claim 6 , the operation further comprising:
constructing the graph based on a plurality of signatures for a plurality of words;
selecting, for each respective subgraph in the graph, a respective pivot word; and
generating, based on the graph, a mapping between the plurality of words and the selected pivot words.
9. The non-transitory computer readable medium of claim 6 , wherein constructing the graph comprises:
assigning a weight to the edge between the first and second nodes based on a number of times the first and second words were assigned matching signatures by the one or more locality-sensitive hashing techniques.
10. The non-transitory computer readable medium of claim 9 , the operation further comprising:
identifying one or more edges in the graph with a weight below a predefined pruning criterion; and
pruning the identified one or more edges.
11. The non-transitory computer readable medium of claim 6 , the operation further comprising:
determining that a third word is not included in the graph;
generating a third signature for the third word using the one or more locality-sensitive hashing techniques; and
generating, based on the third signature, a mapping between the first and third words.
12. The non-transitory computer readable medium of claim 6 , wherein generating the first signature using the one or more locality-sensitive hashing techniques comprises:
generating a set of sub strings contained within the first word;
generating a plurality of hash values for the set of sub strings;
partitioning the plurality of hash values into a plurality of bins;
selecting a representative value for each respective bin of the plurality of bins, wherein the representative value is a smallest value in the respective bin;
filling one or more empty bins of the plurality of bins based on representative values for adjacent bins; and
generating a first signature for the first word based on hashing the representative values.
13. The non-transitory computer readable medium of claim 6 , the operation further comprising:
selecting the first word as a pivot word; and
replacing each instance of the second word in a dataset with the first word.
14. A computer-implemented method, comprising:
generating a first signature for a first word and a second signature for a second word using one or more locality-sensitive hashing techniques;
constructing a graph based on the first and second signatures, comprising:
creating a first node in the graph for the first word;
creating a second node in the graph for the second word; and
upon determining that the first and second signatures match, creating an edge in the graph connecting the first and second nodes; and
generating, based on the graph, a mapping from the first word to the second word.
15. The computer-implemented method of claim 14 , the method further comprising:
generating a first plurality of signatures for the first word using the one or more locality-sensitive hashing techniques a plurality of times;
generating a second plurality of signatures for the second word using the one or more locality-sensitive hashing techniques a plurality of times; and
updating the graph based on the first and second pluralities of signatures.
16. The computer-implemented method of claim 14 , the method further comprising:
constructing the graph based on a plurality of signatures for a plurality of words;
selecting, for each respective subgraph in the graph, a respective pivot word; and
generating, based on the graph, a mapping between the plurality of words and the selected pivot words.
17. The computer-implemented method of claim 14 , wherein constructing the graph comprises:
assigning a weight to the edge between the first and second nodes based on a number of times the first and second words were assigned matching signatures by the one or more locality-sensitive hashing techniques.
18. The computer-implemented method of claim 17 , the method further comprising:
identifying one or more edges in the graph with a weight below a predefined pruning criterion; and
pruning the identified one or more edges.
19. The computer-implemented method of claim 14 , the method further comprising:
determining that a third word is not included in the graph;
generating a third signature for the third word using the one or more locality-sensitive hashing techniques; and
generating, based on the third signature, a mapping between the first and third words.
20. The computer-implemented method of claim 14 , wherein generating the first signature using the one or more locality-sensitive hashing techniques comprises:
generating a set of sub strings contained within the first word;
generating a plurality of hash values for the set of sub strings;
partitioning the plurality of hash values into a plurality of bins;
selecting a representative value for each respective bin of the plurality of bins, wherein the representative value is a smallest value in the respective bin;
filling one or more empty bins of the plurality of bins based on representative values for adjacent bins; and
generating a first signature for the first word based on hashing the representative values.