Topic-based document segmentation
A system and method to identify a document including text relating to a merchant system. The document is segmented into a set of sentences. A first machine-learning model executed by a processing device generates an initial topic segmentation corresponding to the set of sentences. A second machine-learning model is applied to the initial topic segmentation to generate a final topic segmentation corresponding to the document.
1. A method comprising:
identifying a document comprising text relating to a merchant system;
segmenting the document into a set of sentences;
generating, using a first machine-learning model executed by a processing device, an initial topic segmentation corresponding to the set of sentences based on an initial set of probabilities corresponding to each sentence, wherein, for each sentence of the set of sentences, the first machine-learning model generates the initial set of probabilities comprising a first probability corresponding to a first state and a second probability corresponding to a second state, and wherein the first state indicates that the sentence is a new topic and the second state indicates that the sentence is not a new topic; and
generating, using a second machine-learning model applied to the initial topic segmentation, a final topic segmentation corresponding to the document.
2. The method of claim 1 , further comprising:
causing generation of a graphical user interface including the document comprising one or more visual elements corresponding to a set of topics of the final topic segmentation.
3. The method of claim 1 , wherein the document is segmented into the set of sentences by executing a machine-learning model trained to identify a sentence from the text of the document.
4. The method of claim 1 , wherein the first machine-learning model comprises a first hidden markov model trained to identify a vector embedding corresponding to each sentence of the set of sentences.
5. The method of claim 4 , wherein the second machine-learning model is a second hidden markov model trained to identify, for each sentence, a set of updated probabilities based on the vector embedding and the first probability and the second probability.
6. The method of claim 5 , wherein the final topic segmentation comprises a final set of topics determined based on the set of updated probabilities.
7. A system comprising:
a memory to store instructions; and
a processing device operatively coupled to the memory, the processing device to execute the instructions to perform operation comprising:
identifying a document comprising text relating to a merchant system;
segmenting the document into a set of sentences;
generating, using a first machine-learning model, an initial topic segmentation corresponding to the set of sentences based on an initial set of probabilities corresponding to each sentence, wherein, for each sentence of the set of sentences, the first machine-learning model generates the initial set of probabilities comprising a first probability corresponding to a first state and a second probability corresponding to a second state, and wherein the first state indicates that the sentence is a new topic and the second state indicates that the sentence is not a new topic; and
generating, using a second machine-learning model applied to the initial topic segmentation, a final topic segmentation corresponding to the document.
8. The system of claim 7 , the operations further comprising causing generation of a graphical user interface including the document comprising one or more visual elements corresponding to a set of topics of the final topic segmentation.
9. The system of claim 7 , wherein the document is segmented into the set of sentences by executing a machine-learning model trained to identify a sentence from the text of the document.
10. The system of claim 7 , wherein the first machine-learning model comprises a first hidden markov model trained to identify a vector embedding corresponding to each sentence of the set of sentences.
11. The system of claim 10 , wherein the second machine-learning model is a second hidden markov model trained to identify, for each sentence, a set of updated probabilities based on the vector embedding and the first probability and the second probability.
12. The system of claim 11 , wherein the final topic segmentation comprises a final set of topics determined based on the set of updated probabilities.
13. A non-transitory computer readable storage medium having instructions that, if executed by a processing device, cause the processing device to perform operations comprising:
identifying a document comprising text relating to a merchant system; segmenting the document into a set of sentences;
generating, using a first machine-learning model executed by a processing device, an initial topic segmentation corresponding to the set of sentences based on an initial set of probabilities corresponding to each sentence, wherein, for each sentence of the set of sentences, the first machine-learning model generates the initial set of probabilities comprising a first probability corresponding to a first state and a second probability corresponding to a second state, and wherein the first state indicates that the sentence is a new topic and the second state indicates that the sentence is not a new topic; and
generating, using a second machine-learning model applied to the initial topic segmentation, a final topic segmentation corresponding to the document.
14. The non-transitory computer readable storage medium of claim 13 , the operations further comprising causing generation of a graphical user interface including the document comprising one or more visual elements corresponding to a set of topics of the final topic segmentation.
15. The non-transitory computer readable storage medium of claim 13 , the operations further comprising executing a search algorithm to identify the document in view of a search query.
16. The non-transitory computer readable storage medium of claim 13 , the operations further comprising:
generating, by the first machine-learning model, a first set of probabilities corresponding to the set of sentences; and
for each sentence, identifying, by the second machine-learning model comprising a trained second hidden markov model, a set of updated probabilities based on a vector embedding corresponding to each sentence and the first probability and the second probability corresponding to each sentence.