Scalable multilingual named-entity recognition
Software on a website serves a user of an online content aggregation service a first article that the user views. The software extracts named entities from the first article using a named-entity recognizer. The named-entity recognizer uses a sequence of word embeddings as inputs to a conditional random field (CRF) tool to assign labels to each of the word embeddings. Each of the word embeddings is associated with a word in the first article and is trained using an entire topical article from a corpus of topical articles as a context for the word. The software then creates rankings for articles ingested by the content aggregation service based at least in part on the named entities and serves the user a second article using the rankings.
1. A method, comprising operations of:
serving a user of an online content aggregation service a first article of a corpus of topical articles;
responsive to the user viewing the first article, extracting named entities from the first article using a named-entity recognizer, wherein the named-entity recognizer is configured to use a trained distributional-semantic model process that trains on each word in the first article using an entire topical article from the corpus of topical articles as a context for the each word, and the training produces word embeddings that are provided as inputs to a conditional random field (CRF) tool to estimate a conditional probability of values on labels based upon values assigned to a word sequence in the first article and assign labels to concatenated ch of the word embeddings using BILOU (Beginning, In, Last, Outside, Unique) indicators based on the word sequence in the first article, wherein the assigning labels comprises labelling a first word of the word sequence as a beginning word of a person entity, labelling a second word of the word sequence as in the person entity, labelling a third word of the word sequence as a last word of the person entity, labelling a fourth word of the word sequence as outside of an entity, labelling a fifth word of the word sequence as a beginning word of a location entity, labelling a sixth word of the word sequence as a last word of the location entity, and labelling a seventh word as a unique entity, wherein the labeling is language independent and does not use gazetteers;
determining that one or more content items have been viewed by one or more connections of the user on a social network;
creating rankings for articles ingested by the online content aggregation service based at least in part on (i) the one or more content items that have been viewed by the one or more connections of the user and (ii) the named entities associated with the word embeddings having the labels that include one or more BILOU indicators;
generating for the user a content stream, comprising a second article of the corpus of topical articles, based on the rankings; and
serving the content stream to the user, wherein each of the operations is performed by one or more processors.
2. The method of claim 1 , wherein the named-entity recognizer extracts one or more first named entities in a first language and one or more second entities in a second language.
3. The method of claim 1 , wherein each of the word embeddings is a word vector.
4. The method of claim 1 , wherein the trained distributional-semantic model process is a Word2Vec process used for generating each of the word embeddings.
5. The method of claim 1 , wherein each named entity is a person, location, or organization.
6. The method of claim 1 , wherein a sequence of the word embeddings is based on a contextual window that is an integer and that includes a sub-sequence of words to a left of a word in the first article and a sub-sequence of words to a right of the word in the first article.
7. The method of claim 1 , wherein the corpus is in a particular language, and wherein selection of the corpus in the particular language enables use of the language independent labeling in the particular language.
8. One or more non-transitory computer-readable media persistently storing a program, wherein the program, when executed, instructs a processor to:
serve a user of an online content aggregation service a first article of a corpus of topical articles;
responsive to the user viewing the first article, extract named entities from the first article using a named-entity recognizer, wherein the named-entity recognizer uses a sequence of word embeddings as inputs to a conditional random field (CRF) tool to estimate a conditional probability of values on labels based upon values assigned to a word sequence in the first article and assign labels to concatenated word embeddings using BILOU (Beginning, In, Last, Outside, Unique) indicators based on the word sequence in the first article, wherein the labeling is language independent and does not use gazetteers, and wherein each of the word embeddings is associated with a word in the first article and is trained using an entire topical article from the corpus of topical articles as a context for the word, wherein the assigning labels comprises at least one of labelling a first word of the word sequence as a beginning word of a person entity, labelling a second word of the word sequence as in the person entity, labelling a third word of the word sequence as a last word of the person entity, labelling a fourth word of the word sequence as outside of an entity, labelling a fifth word of the word sequence as a beginning word of a location entity, labelling a sixth word of the word sequence as a last word of the location entity, or labelling a seventh word as a unique entity;
determine that one or more content items have been viewed by one or more connections of the user on a social network;
create rankings for articles ingested by the online content aggregation service based at least in part on (i) the one or more content items that have been viewed by the one or more connections of the user and (ii) the named entities associated with the sequence of word embeddings having the labels that include one or more BILOU indicators;
generate for the user a content stream, comprising a second article of the corpus of topical articles, based on the rankings; and
serve the content stream to the user.
9. The non-transitory computer-readable media of claim 8 , wherein the named-entity recognizer does not use clustering techniques.
10. The non-transitory computer-readable media of claim 8 , wherein each of the word embeddings is a word vector.
11. The non-transitory computer-readable media of claim 8 , wherein each of the word embeddings is generated using Word2Vec functionality that is a trained distributional-semantic model process.
12. The non-transitory computer-readable media of claim 8 , wherein each named entity is a person, location, or organization.
13. The non-transitory computer-readable media of claim 8 , wherein the sequence of word embeddings is based on a contextual window that is an integer and that includes a sub-sequence of words to the left of a word in the first article and a sub-sequence of words to the right of the word in the first article.
14. The non-transitory computer-readable media of claim 8 , wherein the corpus is an online encyclopedia in a particular language, and wherein selection of the corpus in the particular language enables use of the language independent labeling in the particular language.
15. A method, comprising operations of:
serving a user of an online content aggregation service a first article of a corpus of topical articles;
responsive to the user viewing the first article, extracting named entities from the first article using a named-entity recognizer, wherein the named-entity recognizer uses a sequence of word vectors as inputs to a conditional random field (CRF) tool to estimate a conditional probability of values on labels based upon values assigned to a word sequence in the first article and assign labels to concatenated word vectors using BILOU (Beginning, In, Last, Outside, Unique) indicators based on the word sequence in the first article, wherein each of the word vectors is associated with a word in the first article and is trained using an entire topical article from the corpus of topical articles as a context for the word;
determining that one or more content items have been viewed by one or more connections of the user on a social network;
creating rankings for articles ingested by the online content aggregation service based at least in part on (i) the one or more content items that have been viewed by the one or more connections of the user and (ii) the named entities associated with the sequence of word vectors having the labels that include one or more BILOU indicators;
generating for the user a content stream, comprising a second article of the corpus of topical articles, based on the rankings; and
serving the content stream to the user, wherein each of the operations is performed by one or more processors;
wherein the sequence of word vectors is based on a contextual window that is an integer and that includes a sub-sequence of words to a left of a word in the first article and a sub-sequence of words to a right of the word in the first article.
16. The method of claim 15 , wherein the corpus is in a particular language.
17. The method of claim 15 , wherein the corpus is in a particular language, and wherein selection of the corpus in the particular language enables use of the language independent labeling in the particular language.