Method for citation identification

1 . A computer-implemented method for generating a citation database from a corpus of documents stored in a computer readable memory, wherein the corpus of documents includes at least ten million documents, the method comprising for each of the documents in the corpus of documents, by executing instructions stored on a processor readable media of at least one computing device:

searching, by the at least one computing device, the document for an entity identifier corresponding to an entity and, if an instance of the entity identifier is detected in the document, determining a portion of the document around the instance of the entity identifier as a target text, wherein the entity is associated with a product catalogue, the product catalogue comprising a plurality of product identifiers;

automatically applying, by the at least one computing device, a first regular expression to the target text, wherein the first regular expression is configured to match one or more of the product identifiers;

identifying, by the at least one computing device, one or more tokens from the target text that match the first regular expression; and

determining, by the at least one computing device, if any of the one or more tokens corresponds to a product identifier from the product catalogue and, if so, which product identifier;

if a product identifier from the product catalogue is determined to be cited in the target text, automatically adding, by the at least one computing device, an entry to a citation database linking the document and the product identifier;

wherein determining if any of the one or more tokens corresponds to a product identifier comprises an iterative process, each iteration of the iterative process including:

determining a set of prefixes for the plurality of product identifiers, each prefix having a predetermined number of characters;

generating a set of prefix regular expressions corresponding to the set of prefixes;

applying the prefix regular expressions to the one or more tokens, and keeping only the prefix regular expressions that return a match;

wherein for each subsequent iteration, the predetermined number of characters is increased by one, and the set of prefixes is determined to include only prefixes that match a prefix regular expression kept in the previous iteration; and

wherein, a token is determined to correspond to a product identifier if a prefix regular expression matches the token, and if the predetermined number of characters corresponds to a number of characters in the product identifier.

2 . A method according to claim 1 , wherein the searching for the entity identifier in the document comprises applying a second regular expression to the document.

3 . A method according to claim 2 , wherein there are multiple entity identifiers corresponding to the entity, and the second regular expression is configured to match all of the multiple entity identifiers.

4 . A method according to claim 1 , further comprising generating the first regular expression.

5 . A method according to claim 4 , wherein generating the first regular expression comprises obtaining, from a memory, by the at least one computing device, a regular expression code associated with the product catalogue, and basing a first component of the first regular expression on the obtained regular expression code.

6 . A method according to claim 5 wherein the at least one product identifiers were determined to not match the regular expression code.

7 . A method according to claim 4 , wherein generating the first regular expression comprises generating, by the at least one computing device, using at least one of the plurality of product identifiers, a second component of the first regular expression, the second component being configured to match the at least one product identifiers.

8 . A method according to claim 7 wherein generating the second component of the first regular expression comprises parsing by the at least one computing device, the at least one product identifiers into a tree, and the at least one computing device is configured for traversing the tree using a recursive walk algorithm to generate the second component.

9 . A method according to claim 4 , further comprising storing the first regular expression in a memory.

10 . A method according to claim 1 further comprising, for each of the one or more tokens determined to correspond to a product identifier:

determining, by the at least one computing device, a risk factor for the corresponding product identifier, wherein the risk factor is associated with a risk of wrongly identifying the product identifier in the target text;

if a risk factor is determined by the at least one computing device, generating a third regular expression configured to match the corresponding product identifier, by the at least one computing device, wherein the third regular expression includes an identifier component configured to match the product identifier, and a context component that is configured to match a predetermined context;

applying, by at least one computing device, the third regular expression to the target text; and

if a match with the third regular expression is found in the target text, determining, by the at least one computing device, that the corresponding product identifier is cited in the target text.

11 . A method according to claim 10 , wherein the third regular expression comprises an identifier component configured to match the product identifier, and a context component that is configured to match a predetermined context.

12 . A method according to claim 1 , wherein the first regular expression is configured to match all of the plurality of product identifiers.

13 . A computer-implemented system comprising a processor and a memory storing instructions which, when executed by the processor, cause the processor to carry out the method of claim 1 .

14 . A computer-readable non-volatile storage medium comprising instructions stored therein which, when executed by a processor, cause the processor to carry out the method of claim 1 .

15 . The method according to claim 1 , wherein the first regular expression is obtained by automatically generating, by the at least one computing device, at least a portion of the first regular expression by iterating over the plurality of product identifiers and constructing a regular expression that matches all of the product identifiers.

16 . A computer-implemented method for generating a product citation database from a corpus of documents, the method comprising:

searching, by at least one processor configured by code, for each electronically accessible document within the corpus of documents, for an entity identifier corresponding to an entity and, if an instance of the entity identifier is detected in the document, determining, by at least one processor configured by code, a portion of the document around the instance of the entity identifier as a target text, wherein the entity is associated with a product catalogue, the product catalogue comprising a plurality of product identifiers;

identifying, by the at least one processor, a risk factor associated with a product identifier in the document, wherein the risk factor indicates a likelihood of the product identifier being identified in the document is a false positive in the target text;

generating, by the at least one processor, a first regular expression that includes: (i) an identifier component matching the product identifier; and (ii) a context component matching a predetermined character string adjacent to the identifier component, wherein the context component is obtained by retrieving the context component from a database of a plurality of context components;

automatically applying the first regular expression to the target text by at least one processor configured by code, wherein the first regular expression is configured to match one or more of the plurality of product identifiers using the identifier component and the context component; and

if a product identifier from the plurality of product identifiers is determined to be cited in the target text;

generating a citation database file by at least one processor configured by code, where the citation database does not exist; and

adding an entry to the citation database by at least one processor configured by code, linking the document and the product identifier.

17 . A computer-implemented method for searching content within a database, the method operating within a processor configured by code comprising:

receiving a search query for a product,

performing a search on the citation database wherein the citation database was generated by the method of claim 16 , and

returning one or more product identifiers from the citation database based on the search query, wherein the one or more product identifiers are ranked based on a number of documents linked with each product identifier in the citation database.