Context-based keyphrase extraction from input text
Described herein are systems, methods, and other techniques for extracting one or more keyphrases from an input text. The input text may include a plurality of words. A plurality of token-level attention matrices may be generated using a transformer-based machine learning model. The plurality of token-level attention matrices may be converted into a plurality of word-level attention matrices. A set of candidate phrases may be identified from the plurality of words based on the plurality of word-level attention matrices. The one or more keyphrases may be selected from the set of candidate phrases.
1 . A computer-implemented method for extracting one or more keyphrases from an input text, the computer-implemented method comprising:
receiving the input text having a plurality of words;
providing the plurality of words to a transformer-based machine learning model;
generating, using the transformer-based machine learning model, a plurality of token-level attention matrices that indicate token-level context-based relationship values between tokens in the plurality of words;
converting the plurality of token-level attention matrices into a plurality of word-level attention matrices that indicate word-level context-based relationship values between the plurality of words;
identifying, according to the word-level context-based relationship values defined by the plurality of word-level attention matrices, a set of candidate phrases from the plurality of words, wherein each of the set of candidate phrases includes one or more words from the plurality of words; and
selecting the one or more keyphrases from the set of candidate phrases by:
sorting the set of candidate phrases according to first token positions within the set of candidate phrases;
determining, based on sorting the set of candidate phrases, an amount of candidate phrase overlap between a first candidate phrase and a second candidate phrase among the set of candidate phrases; and
generating a keyphrase score by combining a first score for the first candidate phrase with a second score for the second candidate phrase according to the amount of candidate phrase overlap.
2 . The computer-implemented method of claim 1 , further comprising:
generating a plurality of clusters based on the plurality of word-level attention matrices, wherein each of the plurality of clusters is generated based on the word-level context-based relationship values, and wherein the set of candidate phrases are identified based on the plurality of clusters.
3 . The computer-implemented method of claim 2 , wherein the plurality of clusters are generated by determining which of the word-level context-based relationship values are greater than a threshold.
4 . The computer-implemented method of claim 1 , further comprising:
preprocessing the input text by dividing the plurality of words into a plurality of segments, wherein each of the plurality of segments is separately provided to the transformer-based machine learning model.
5 . The computer-implemented method of claim 1 , wherein the transformer-based machine learning model includes a plurality of heads, wherein each of the plurality of heads (i) includes a set of attention weights and (ii) generates one of the plurality of token-level attention matrices.
6 . The computer-implemented method of claim 1 , further comprising:
determining that the amount of candidate phrase overlap satisfies an overlap threshold; and
combining the first candidate phrase and the second candidate phrase based on determining that the amount of candidate phrase overlap satisfies the overlap threshold.
7 . The computer-implemented method of claim 1 , further comprising:
utilizing a duplicate detector to detect that a third candidate phrase is a duplicate of the first candidate phrase; and
based on detecting that the third candidate phrase is a duplicate of the first candidate phrase:
increasing the keyphrase score associated with the first candidate phrase; and
removing the third candidate phrase from the set of candidate phrases.
8 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving an input text having a plurality of words;
providing the plurality of words to a transformer-based machine learning model;
generating, using the transformer-based machine learning model, a plurality of token-level attention matrices that indicate token-level context-based relationship values between tokens in the plurality of words;
converting the plurality of token-level attention matrices into a plurality of word-level attention matrices that indicate word-level context-based relationship values between the plurality of words;
identifying, according to the word-level context-based relationship values defined by the plurality of word-level attention matrices, a set of candidate phrases from the plurality of words, wherein each of the set of candidate phrases includes one or more words from the plurality of words; and
selecting one or more keyphrases from the set of candidate phrases by:
sorting the set of candidate phrases according to first token positions within the set of candidate phrases;
determining, based on sorting the set of candidate phrases, an amount of candidate phrase overlap between a first candidate phrase and a second candidate phrase among the set of candidate phrases; and
generating a keyphrase score by combining a first score for the first candidate phrase with a second score for the second candidate phrase according to the amount of candidate phrase overlap.
9 . The non-transitory computer-readable medium of claim 8 , wherein the operations further comprise:
generating a plurality of clusters based on the plurality of word-level attention matrices, wherein each of the plurality of clusters is generated based on the word-level context-based relationship values, and wherein the set of candidate phrases are identified based on the plurality of clusters.
10 . The non-transitory computer-readable medium of claim 9 , wherein the plurality of clusters are generated by determining which of the word-level context-based relationship values are greater than a threshold.
11 . The non-transitory computer-readable medium of claim 8 , further comprising:
preprocessing the input text by dividing the plurality of words into a plurality of segments, wherein each of the plurality of segments is separately provided to the transformer-based machine learning model.
12 . The non-transitory computer-readable medium of claim 8 , wherein the transformer-based machine learning model includes a plurality of heads, wherein each of the plurality of heads (i) includes a set of attention weights and (ii) generates one of the plurality of token-level attention matrices.
13 . The non-transitory computer-readable medium of claim 8 , wherein the transformer-based machine learning model is a Bidirectional Encoder Representations from Transformers (BERT) model.
14 . The non-transitory computer-readable medium of claim 8 , wherein selecting the one or more keyphrases from the set of candidate phrases includes:
determining a set of scores for the set of candidate phrases based on the word-level context-based relationship values between the plurality of words.
15 . A system comprising,
one or more processors; and
a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving an input text having a plurality of words;
providing the plurality of words to a transformer-based machine learning model;
generating, using the transformer-based machine learning model, a plurality of token-level attention matrices that indicate token-level context-based relationship values between tokens in the plurality of words;
converting the plurality of token-level attention matrices into a plurality of word-level attention matrices that indicate word-level context-based relationship values between the plurality of words;
identifying, according to the word-level context-based relationship values defined by the plurality of word-level attention matrices, a set of candidate phrases from the plurality of words, wherein each of the set of candidate phrases includes one or more words from the plurality of words; and
selecting one or more keyphrases from the set of candidate phrases by:
sorting the set of candidate phrases according to first token positions within the set of candidate phrases;
determining, based on sorting the set of candidate phrases, an amount of candidate phrase overlap between a first candidate phrase and a second candidate phrase among the set of candidate phrases; and
generating a keyphrase score by combining a first score for the first candidate phrase with a second score for the second candidate phrase according to the amount of candidate phrase overlap.
16 . The system of claim 15 , wherein the operations further comprise:
generating a plurality of clusters based on the plurality of word-level attention matrices, wherein each of the plurality of clusters is generated based on the word-level context-based relationship values, and wherein the set of candidate phrases are identified based on the plurality of clusters.
17 . The system of claim 16 , wherein the plurality of clusters are generated by determining which of the word-level context-based relationship values are greater than a threshold.
18 . The system of claim 15 , further comprising:
preprocessing the input text by dividing the plurality of words into a plurality of segments, wherein each of the plurality of segments is separately provided to the transformer-based machine learning model.
19 . The system of claim 15 , wherein the transformer-based machine learning model includes a plurality of heads, wherein each of the plurality of heads (i) includes a set of attention weights and (ii) generates one of the plurality of token-level attention matrices.
20 . The system of claim 15 , wherein selecting the one or more keyphrases from the set of candidate phrases includes:
determining a set of scores for the set of candidate phrases based on the word-level context-based relationship values between the plurality of words.