Token matching in large document corpora

1. A method comprising:

receiving, by a server, a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;

automatically constructing, by the server, a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;

automatically populating, by the server, the probabilistic data representation model with a data representation of the plurality of entities;

receiving, by the server, a text corpus that is segmented into tokens; and

automatically matching, by the server, each token in the text corpus against the populated probabilistic data representation model.

2. The method of claim 1 , wherein, with respect to each BF pair indexed i, a first BF in the BF pair is populated with first i tokens from the plurality of entities having at least i+1 tokens, and a second BF in the BF pair in populated with all from the plurality of entities having exactly i tokens.

3. The method of claim 1 , wherein the matching further comprises sequentially querying each BF pair in order of the indexing from 1 to n, to determine a match.

4. The method of claim 1 , wherein each BF pair has a specified number of hash functions, each specified number being equal to an optimal number calculated to minimize an error probability.

5. The method of claim 1 , wherein each BFP pair has a specified number of hash functions, each specified number being less than an optimal number calculated to minimize an error probability.

6. The method of claim 1 , wherein each BF pair has a specified number of hash functions, each specified number being less than an optimal number calculated to minimize an error probability, the specified number further decreasing as the indexing of the BF pairs increases.

7. A computing device comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions for performing a method of token matching;

a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:

receive a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;

automatically construct a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;

automatically populate the probabilistic data representation model with a data representation of the plurality of entities;

receive a text corpus that is segmented into tokens; and

automatically match each token in the text corpus against the populated probabilistic data representation model.

8. The computing device of claim 7 , wherein, with respect to each BF pair indexed i, a first BF in the BF pair is populated with first i tokens from the plurality of entities having at least i+1 tokens, and a second BF in the BF pair in populated with all from the plurality of entities having exactly i tokens.

9. The computing device of claim 7 , the machine executable code further causing the processor to: wherein said matching comprises sequentially query each BF pair in order of the indexing from 1 to n, to determine a match.

10. The computing device of claim 7 , wherein each BF pair has a specified number of hash functions calculated to minimize an error probability.

11. The computing device of claim 7 , wherein BF pair has a specified number of hash functions, each specified number being less than an optimal number calculated to minimize an error probability.

12. The computing device of claim 7 , wherein each BF pair has a specified number of hash functions, each specified number being less than an optimal number calculated to minimize an error probability, the specified number further decreasing as the indexing of the BF pairs increases.

13. A non-transitory machine readable medium having stored thereon instructions for performing a method comprising machine executable code which when executed by at least one machine, causes the machine to:

receive a dictionary comprising a plurality of entities, wherein each entity has a length of between 1 and n tokens;

automatically construct a probabilistic data representation model comprising n Bloom filter (BF) pairs, wherein each BF pair is indexed from 1 to n;

automatically populate the probabilistic data representation model with a data representation of the plurality of entities;

receive a text corpus that is segmented into tokens; and

automatically match each token in the text corpus against the populated probabilistic data representation model.

14. The non-transitory machine readable medium of claim 13 , wherein, with respect to each BF pair indexed i, a first BF in BF pair is populated with first i tokens from the plurality of entities having at least i+1 tokens, and a second BF in the BF pair in populated with all from the plurality of entities having exactly i tokens.

15. The non-transitory machine readable medium of claim 13 , further comprising machine executable code that causes the machine to: sequentially query each BF pair in order of the indexing from 1 to n, to determine a match.

16. The non-transitory machine readable medium of claim 13 , wherein each BF pair has a specified number of hash functions, each specified number being equal to an optimal number calculated to minimize an error probability.

17. The non-transitory machine readable medium of claim 13 , wherein each BF pair has a specified number of hash functions each being less than an optimal number calculated to minimize an error probability.

18. The non-transitory machine readable medium of claim 13 , wherein each BF pair has a specified number of hash functions, each specified number being less than an optimal number calculated to minimize an error probability, the specified number further decreasing as the indexing of the BF pairs increases.