Search system and corresponding method
There is provided a search system comprising a statistical model trained on text associated with a piece of content. The text associated with the piece of content is drawn from a plurality of different data sources. The system is configured to receive text input and generate using the statistical model an estimate of the likelihood that the piece of content is relevant given the text input. A corresponding method is also provided.
1. A system comprising:
one or more processors; and
one or more storage medium comprising program instructions, wherein execution of the program instructions by the one or more processors causes the one or more processors to:
receive text input comprising one or more n-grams;
identify a content type indicated by the one or more n-grams of the text input using a classifier model, wherein the classifier model is trained with a set of text segments and associated content type labels for each text segment, and wherein the content type identifies a subject matter of the one or more n-grams of the text input, the subject matter being representable in a plurality of representation formats;
in response to identifying the content type for the text input, select a single statistical model from among a set of statistical models, wherein each statistical model of the set of statistical models is associated with a respective content type, based on datasets of content used to train the respective statistical model and wherein the selection is based on the identified content type, indicated by the one or more n-grams of the text input, matching the content type of the statistical model;
generate, with the selected statistical model, a relevance estimate for a content item in relation to the one or more n-grams of the text input, wherein the selected statistical model is trained with text associated with the content item and a relevance value in relation to the text and the content item, and wherein the text associated with the content item is drawn from a plurality of different data sources; and
output the content item as a content suggestion when the relevance estimate of the content item exceeds a predetermined relevance threshold.
2. The system of claim 1 , wherein the statistical model is trained on one or more features extracted from the text associated with the content item, and wherein execution of the program instructions by the one or more processors causes the one or more processors to:
extract the one or more features from the text input, and
query the statistical model with each of the one or more features of the text input to generate the relevance estimate for the content item in relation to the text input.
3. The system of claim 1 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
generate a respective relevance estimate for each content item of a plurality of content items in relation to the text input using a statistical model, associated with each content item, selected from a plurality of statistical models, and trained on text associated with pieces of content items; and
rank the plurality of content items by the respective relevance estimate and determine one or more of the most relevant content items based on the relevance estimate.
4. The system of claim 3 , wherein execution of the program instructions by the one or more processors causes the one or more processors to output at least one representation of each of the one or more most relevant content items.
5. The system of claim 3 , wherein at least one content item is associated with a particular entity and wherein execution of the program instructions by the one or more processors causes the one or more processors to output at least one representation of one or more entities associated with the one or more most relevant content items.
6. The system of claim 3 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
classify, by content type, each of the plurality of statistical models trained on text;
classify the text input as a content type; and
determine a subset of the plurality of statistical models which are classified the same content type as the content type of the text input.
7. The system of claim 1 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
compare a portion of the text input to a first language model trained on natural language text;
compare the portion of the text input to a second language model trained on conversational text; and
classify the portion of the text input as natural language or conversational.
8. The system of claim 1 , wherein the plurality of representation formats include at least one of a text file, an image, a video clip, or a URL.
9. A method for determining whether a content item is relevant to text input, the method comprising:
receiving text input comprising one or more n-grams;
identifying a content type indicated by the one or more n-grams of the text input using a classifier model, wherein the classifier model is trained with a set of text segments and associated content type labels for each text segment, and wherein the content type identifies a subject matter of the one or more n-grams of the text input, the subject matter being representable in a plurality of representation formats;
in response to identifying the content type for the text input, selecting a single statistical model from among a set of statistical models, wherein each statistical model of the set of statistical models is associated with a respective content type, based on datasets of content used to train the respective statistical model and wherein the selection is based on the identified content type, indicated by the one or more n-grams of the text input matching the content type of the statistical model;
generating, with the selected statistical model, a relevance estimate for a content item in relation to the one or more n-grams of the text input, wherein the selected statistical model is trained with text associated with the content item and a relevance value in relation to the text and the content item, and wherein the text associated with the content item is drawn from a plurality of different data sources; and
outputting the content item as a content suggestion when the relevance estimate of the content item exceeds a predetermined relevance threshold.
10. The method of claim 9 , further comprising:
extracting, using one or more processors, one or more features from the text input; and
querying, using one or more processors, the statistical model with each of the one or more feature of the text input to generate the relevance estimate for the content item in relation to the text input.
11. A non-transitory computer-readable medium containing program instructions for determining whether a content item is relevant to text input, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to:
receive text input comprising one or more n-grams;
identify a content type indicated by the one or more n-grams of the text input using a classifier model, wherein the classifier model is trained with a set of text segments and associated content type labels for each text segment, and wherein the content type identifies a subject matter of the one or more n-grams of the text input, the subject matter being representable in a plurality of representation formats;
in response to identifying the content type for the text input, select a single statistical model from among a set of statistical models, wherein each statistical model of the set of statistical models is associated with a respective content type, based on datasets of content used to train the respective statistical model and wherein the selection is based on the identified content type, indicated by the one or more n-grams of the text input, matching the content type of the statistical model;
generate, with the selected statistical model, a relevance estimate for a content item in relation to the one or more n-grams of the text input, wherein the selected statistical model is trained with text associated with the content item and a relevance value in relation to the text and the content item, and wherein the text associated with the content item is drawn from a plurality of different data sources; and
output the content item as a content suggestion when the relevance estimate of the content item exceeds a predetermined relevance threshold.
12. The non-transitory computer-readable medium of claim 11 , wherein the text input is not input into the computer system by a user.
13. The non-transitory computer-readable medium of claim 11 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
receive non-textual evidence; and
generate using a statistical model that is trained on non-textual data the relevance estimate for the content item in relation to the non-textual evidence.
14. The non-transitory computer-readable medium of claim 11 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
classify, by content type, each of a plurality of statistical models trained on text;
classify the text input as a content type; and
determine a subset of the plurality of statistical models which are classified the same content type as the content type of the text input.
15. The non-transitory computer-readable medium of claim 14 , wherein the selected statistical model is trained on one or more features extracted from the text associated with the content item, wherein execution of the program instructions by the one or more processors causes the one or more processors to:
extract the one or more features from the text input, and
query each of the statistical models of the subset of statistical models with each of the one or more features of the text input to generate a relevance estimate for each content item associated with each statistical model of the subset of statistical models in relation to the text input.
16. The non-transitory computer-readable medium of claim 11 , wherein execution of the program instructions by the one or more processors causes the one or more processors to:
compare a portion of the text input to a first language model trained on natural language text;
compare the portion of the text input to a second language model trained on conversational text; and
classify the portion of the text input as natural language or conversational.
17. The non-transitory computer-readable medium of claim 16 , wherein execution of the program instructions by the one or more processors causes the one or more processors to discard the portion of the text input if it is classified as conversational.