Token matching in large document corpora
A method comprising receiving a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens; constructing a probabilistic data representation model comprising n Bloom filter (BF) pairs indexed from 1 to n; populating said probabilistic data representation model with a data representation of said entities, wherein, with respect to each BF pair indexed i: (i) a first BF is populated with the first i tokens of all said entities having at least i+1 tokens, and (ii) a second BF in populated with all said entities having exactly i tokens; receiving a text corpus, wherein said text corpus is segmented into tokens; and automatically matching each token in said text corpus against said populated probabilistic data representation model, wherein said matching comprises sequentially querying each said BF pair in the order of said indexing, to determine a match.
1. A method comprising:
in a populating stage, receiving a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;
automatically constructing a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;
automatically populating said probabilistic data representation model with a data representation of said entities, wherein, with respect to each BF pair indexed i:
(i) a first BF in said BF pair is populated with the first i tokens of all said entities having at least i+1 tokens, and
(ii) a second BF in said BF pair in populated with all said entities having exactly i tokens;
in a matching stage, receiving a text corpus, wherein said text corpus is segmented into tokens; and
automatically matching each token in said text corpus against said populated probabilistic data representation model, wherein said matching comprises sequentially querying each said BF pair in the order of said indexing, to determine a match.
2. The method of claim 1 , wherein each said BF pair has a specified number of hash functions, and wherein said specified number is equal to an optimal number calculated to minimize an error probability.
3. The method of claim 2 , wherein said specified number is less than said optimal number.
4. The method of claim 3 , wherein said specified number further decreases when said indexing of said BF pairs increases.
5. A system comprising:
at least one hardware processor; and
a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
in a populating stage, receive a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;
automatically construct a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;
automatically populate said probabilistic data representation model with a data representation of said entities, wherein, with respect to each BF pair indexed i:
(i) a first BF in said BF pair is populated with the first i tokens of all said entities having at least i+1 tokens, and
(ii) a second BF in said BF pair in populated with all said entities having exactly i tokens;
in a matching stage, receive a text corpus, wherein said text corpus is segmented into tokens; and
automatically match each token in said text corpus against said populated probabilistic data representation model, wherein said matching comprises sequentially querying each said BF pair in the order of said indexing, to determine a match.
6. The system of claim 5 , wherein each said BF pair has a specified number of hash functions, wherein said specified number is equal to an optimal number calculated to minimize an error probability.
7. The system of claim 6 , wherein said specified number is less than said optimal number.
8. The system of claim 7 , wherein said specified number further decreases when said indexing of said BF pairs increases.
9. A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to:
in a populating stage, receive a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;
automatically construct a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;
automatically populate said probabilistic data representation model with a data representation of said entities, wherein, with respect to each BF pair indexed i:
(i) a first BF in said BF pair is populated with the first i tokens of all said entities having at least i+1 tokens, and
(ii) a second BF in said BF pair in populated with all said entities having exactly i tokens;
in a matching stage, receive a text corpus, wherein said text corpus is segmented into tokens; and
automatically match each token in said text corpus against said populated probabilistic data representation model, wherein said matching comprises sequentially querying each said BF pair in the order of said indexing, to determine a match.
10. The computer program product of claim 9 , wherein each said BF pair has a specified number of hash functions, wherein said specified number is equal to an optimal number calculated to minimize an error probability.
11. The computer program product of claim 10 , wherein said specified number is less than said optimal number.
12. The computer program product of claim 11 , wherein said specified number further decreases when said indexing of said BF pairs increases.