Automatically linking text to concepts in a knowledge base

1. A computer program product for automatically linking text to concepts in a knowledge base, the computer program product comprising:

a non-transitory storage medium readable by a processing circuit and storing instructions for execution by the processing circuit to perform a method comprising:

receiving, at a computer system, a plurality of text strings;

building a conceptual index that links the text strings to the knowledge base, the building comprising for each of the text strings:

selecting a plurality of data sources that correspond to at least a subset of the concepts in the knowledge base, the selecting based on contents of the text string;

calculating, for each of the selected data sources, a probability that the text string is output by a language model built using the selected data source;

calculating a probability that the text string is output by a generic language model that is not related to any particular concept in the knowledge base;

calculating link confidence scores for each of the at least a subset of the concepts based on a differential analysis of the probabilities; and

creating an entry in the conceptual index that includes a link between the text string and one of the concepts in the knowledge base, the creating based at least in part on a link confidence score of the concept being more than a first threshold value away from a prescribed threshold;

generating a conceptual inverted index based on entries in the conceptual index, each entry of the conceptual inverted index corresponding to a different one of the concepts in the knowledge base and comprising pointers to at least a subset of text strings of the plurality of text strings linked to the concept in the conceptual index;

receiving a query from an agent external to the computer system, the query specifying a concept in the knowledge base;

processing the query by the computer system, the processing comprising searching the conceptual inverted index for the concept specified in the query and returning a pointer to a text string in an entry of the conceptual inverted index corresponding to the concept; and

returning a set of documents to the external agent through the use of the conceptual inverted index, based on the received query.

2. The computer program product of claim 1 , wherein the differential analysis compares at least one of:

the probability that the text string is output by a language model built using a data source to the probability that the text string is output by the generic language model; and

the probability that the text string is output by a language model built using a data source to a probability that the text string is output by a language model built using a competing data source.

3. The computer program product of claim 1 , wherein the generic language model is derived from a generic data source not specific to any of the concepts in the knowledge base.

4. The computer program product of claim 1 , wherein the calculating link confidence scores includes comparing the probabilities to a probability that the text string is contained in a generic data source that is not associated with any of the concepts in the knowledge base.

5. The computer program product of claim 1 , wherein the text string is linked to a second one of the concepts in the knowledge base.

6. The computer program product of claim 1 , wherein the link applies to a subset of the text string and the subset is indicated in the link, and words in the subset are not consecutive in the text string.

7. The computer program product of claim 1 , wherein each of the plurality of text strings corresponds to a person and includes a description of their skills, and each of the concepts in the knowledge base is an area of expertise, wherein the query result provides the external agent with a list of possible people having a specified area of expertise.

8. The computer program product of claim 1 , wherein each of the text strings have a version number and the method further comprises periodically, by a garbage collection mechanism, deleting links in the conceptual index to text strings having invalid version numbers.

9. A system for automatically linking text to concepts in a knowledge base, the system comprising:

a memory having computer readable computer instructions; and

one or more processors for executing the computer readable instructions, the computer readable instructions including:

receiving a plurality of text strings;

building a conceptual index that links the text strings to the knowledge base, the building comprising for each of the text strings:

selecting a plurality of data sources that correspond to at least a subset of the concepts in the knowledge base, the selecting based on contents of the text string;

calculating, for each of the selected data sources, a probability that the text string is output by a language model built using the selected data source;

calculating a probability that the text string is output by a generic language model that is not related to any particular concept in the knowledge base;

calculating link confidence scores for each of the at least a subset of the concepts based on a differential analysis of the probabilities; and

creating an entry in the conceptual index that includes a link between the text string and one of the concepts in the knowledge base, the creating based at least in part on a link confidence score of the concept being more than a first threshold value away from a prescribed threshold;

generating a conceptual inverted index based on entries in the conceptual index, each entry of the conceptual inverted index corresponding to a different one of the concepts in the knowledge base and comprising pointers to at least a subset of text strings of the plurality of text strings linked to the concept in the conceptual index;

receiving a query from an agent external to the computer system, the query specifying a concept in the knowledge base;

processing the query, the processing comprising searching the conceptual inverted index for the concept specified in the query and returning a pointer to a text string in an entry of the conceptual inverted index corresponding to the concept; and

returning a set of documents to the external agent through the use of the conceptual inverted index, based on the received query.

10. The system of claim 9 , wherein the differential analysis compares at least one of:

the probability that the text string is output by a language model built using a data source to the probability that the text string is output by the generic language model; and

the probability that the text string is output by a language model built using a data source to a probability that the text string is output by a language model built using a competing data source.

11. The system of claim 9 , wherein the generic language model is derived from a generic data source not specific to any of the concepts in the knowledge base.

12. The system of claim 9 , wherein the calculating link confidence scores includes comparing the probabilities to a probability that the text string is contained in a generic data source that is not associated with any of the concepts in the knowledge base.

13. The system of claim 9 , wherein each of the plurality of text strings corresponds to a person and includes a description of their skills, and each of the concepts in the knowledge base is an area of expertise, wherein the query result provides the external agent with a list of possible people having a specified area of expertise.

14. The system of claim 9 , wherein each of the text strings have a version number and the method further comprises periodically, by a garbage collection mechanism, deleting links in the conceptual index to text strings having invalid version numbers.