Semantic text segmentation based on topic recognition
Systems and methods for semantic temporal segmentation based on topic recognition are disclosed. Text classification and segmentation may be used to index media content for subsequent searches. The method may include using a text classification model to semantically analyze sentences in a text file (such as a transcript) to determine topics with which sentences are associated. The output of the text classification model may be provided to a text segmentation model to enable more-accurate identification of text segments (e.g., paragraphs) within the text file. In some examples, the output of a text segmentation model is provided to a text classification model to enable the text classification model to perform more-accurate classification based on text segments rather than (or in addition to) performing classification on single sentences. The classification of the text segments may be used to assign labels to the text segments to enable subsequent searches based on the labels.
1 . A computer-implemented method, comprising:
obtaining a first set of tokens associated with a first sentence of a transcript and a second set of tokens associated with a second sentence of the transcript;
determining, by a text classification model and based on the first set of tokens, a first probability that a first class is associated with the first sentence;
determining, by the text classification model and based on the second set of tokens, a second probability that the first class is associated with the second sentence;
identifying a first text segment that includes the first sentence and the second sentence based on:
a comparison of a semantic similarity of the first class associated with the first probability to the first class associated with the second probability; and
a comparison of a temporal proximity of a first timestamp associated with the first sentence to a second timestamp associated with the second sentence;
determining a third probability that the first class is associated with the first text segment based on the first set of tokens and the second set of tokens, wherein the third probability is used to adjust a parameter of the text classification model;
determining a label associated with the first text segment based on the third probability;
in response to receiving input indicating a selection of the label, identifying media content corresponding to the first text segment; and
providing the media content for presentation.
2 . The computer-implemented method of claim 1 , wherein the first probability, the second probability, and the third probability are determined by a text classification model comprising a neural network, the method further comprising:
adjusting a parameter of the text classification model based on the third probability.
3 . The computer-implemented method of claim 1 , further comprising:
storing, with the label associated with the first text segment, a first timestamp associated with the first text segment, wherein the first timestamp is determined based on a second timestamp associated with the first sentence, a third timestamp associated with the second sentence, or both.
4 . The computer-implemented method of claim 1 , further comprising:
transcribing media content to generate a plurality of sentences including the first sentence and the second sentence;
tokenizing the first sentence to generate the first set of tokens; and
tokenizing the second sentence to generate the second set of tokens.
5 . The computer-implemented method of claim 1 , wherein determining the first probability comprises:
encoding the first sentence into an embeddings vector using the first set of tokens; and
determining the first probability based on the embeddings vector.
6 . A computer-implemented method, comprising:
generating a first embeddings vector representing semantic content of a first sentence of a transcript and a second embeddings vector representing semantic content of a second sentence of the transcript;
generating an adjusted first embeddings vector based on the first embeddings vector and the second embeddings vector by adjusting the first embeddings vector;
generating an adjusted second embeddings vector based on the first embeddings vector and the second embeddings vector by adjusting the second embeddings vector;
identifying, based on the adjusted first embeddings vector and the adjusted second embeddings vector, a first text segment that includes the first sentence and the second sentence, wherein identifying the first text segment comprises comparing a semantic similarity of a first class associated with the adjusted first embeddings vector and the adjusted second embeddings vector;
determining, based on the first text segment, a probability that the first class is associated with the first text segment;
receiving input indicating a selection of a label;
determining the label is associated with the first text segment based on the probability;
identifying media content corresponding to the first text segment; and
providing the media content for presentation.
7 . The computer-implemented method of claim 6 , further comprising:
transcribing media content to generate a transcript comprising the first sentence and the second sentence.
8 . The computer-implemented method of claim 6 , wherein the first sentence is associated with a first timestamp and the second sentence is associated with a second timestamp, the method further comprising:
determining a timestamp associated with the first text segment based on the first timestamp, the second timestamp, or both; and
storing the label associated with the first text segment with the timestamp associated with the first text segment.
9 . The computer-implemented method of claim 6 , wherein the first embeddings vector is generated by a first text encoder in parallel with the second embeddings vector being generated by a second text encoder.
10 . The computer-implemented method of claim 9 , wherein a fusion-layer transformer adjusts the first embeddings vector and the second embeddings vector based on an output of a global attention mechanism executed on the first embeddings vector and the second embeddings vector.
11 . A system, comprising:
one or more processing systems; and
memory storing instructions that, when executed by the one or more processing systems, cause the system to:
obtain a first set of tokens associated with a first sentence of a transcript and a second set of tokens associated with a second sentence of the transcript;
determine, based on the first set of tokens, a first probability that a first class is associated with the first sentence;
determine, based on the second set of tokens, a second probability that the first class is associated with the second sentence;
identify a first text segment that includes the first sentence and the second sentence based on the first probability, a first timestamp associated with the first sentence, the second probability, and a second timestamp associated with the second sentence; and
determine a third probability that the first class is associated with the first text segment based on the first set of tokens and the second set of tokens, wherein the third probability is used to adjust a parameter of a text classification model;
determining a label associated with the first text segment based on the third probability;
in response to receiving input indicating a selection of the label, identifying media content corresponding to the first text segment; and
providing the media content for presentation.
12 . The system of claim 11 , wherein the first probability, the second probability, and the third probability are determined by the text classification model; and
wherein the instructions parameter of the text classification model is based on the third probability.
13 . The system of claim 11 , wherein the instructions further cause the system to:
store, with the label associated with the first text segment, a first timestamp associated with the first text segment, wherein the first timestamp is determined based on a second timestamp associated with the first sentence a third timestamp associated with the second sentence, or both.
14 . The system of claim 11 , wherein the instructions further cause the system to:
transcribe media content to generate a plurality of sentences including the first sentence and the second sentence;
tokenize the first sentence to generate the first set of tokens; and
tokenize the second sentence to generate the second set of tokens.
15 . The system of claim 11 , wherein determining the first probability comprises:
encoding the first sentence into an embeddings vector using the first set of tokens; and
determining the first probability based on the embeddings vector.
16 . The system of claim 11 , wherein at least one of the first probability or the second probability is determined by the text classification model, wherein the text classification model is a language model trained using a list of predetermined classes.
17 . The computer-implemented method of claim 1 , wherein the first probability and the second probability are determined by the text classification model; and
wherein the first text segment is identified by a text segmentation model.
18 . The computer-implemented method of claim 17 , wherein the text classification model determines the first class is associated with the first sentence based on determining a probability that the sentence is related to the first class exceeds an absolute threshold.
19 . The computer-implemented method of claim 17 , wherein the text classification model determines the first class is associated with the first sentence based on determining a first probability that the sentence is related to the first class exceeds a second probability that the sentence is related to a second class.
20 . The computer-implemented method of claim 17 , wherein the text classification model is a language model trained using a list of predetermined classes.