Text segmentation with two-level transformer and auxiliary coherence modeling
Data is received that encapsulates a document of text. The text is then segmented into a plurality of semantically coherent units using a coherence-aware text segmentation (CATS) machine learning model. Data is then provided that characterizes the segmenting. Related apparatus, systems, techniques and articles are also described.
1. A computer-implemented method comprising:
receiving data encapsulating a document of text;
segmenting the text into a plurality of semantically coherent units using a coherence-aware text segmentation (CATS) machine learning model, the CATS machine learning model being a multi-task learning model that alternatively minimizes a sentence-level segmentation objective and a coherence objective and which differentiates correct sequences of sentences in the document from corrupt sequences of sentences in the document, the CATS model being generated by cross-lingual zero-shot transfer in which a supervised alignment model is used to project target-language vectors from an independently trained embedding space of a target language to a monolingual embedding space of a source language; and
providing data characterizing the segmenting.
2. The method of claim 1 , wherein the CATS machine learning model comprises two hierarchically connected transformer networks.
3. The method of claim 2 , wherein the document of text comprises a sequence of sentences of arbitrary length.
4. The method of claim 2 further comprising:
encoding the text into a plurality of tokens prior to the segmenting.
5. The method of claim 4 , wherein the encoding comprises a concatenation of a pretrained word embedding and a position embedding.
6. The method of claim 4 , wherein a first of the transformer networks is a token-level transformer that encodes sentences from their corresponding tokens.
7. The method of claim 4 , wherein a second of the transformer networks is a sentence-level transformer which receives an output of the first transformer network and generates transformed sentence representations including a representation of a whole sequence of sentences.
8. The method of claim 7 further comprising:
providing the transformed sentence representation to a feed forward segmentation classifier to generate a binary segmentation prediction for each sentence.
9. The method of claim 7 further comprising:
feeding an encoding of the document of text to a neural network to generate a coherence score.
10. The method of claim 1 , wherein providing the data comprises at least one of: causing the data characterizing the segmenting to be displayed in a graphical user interface, loading the data characterizing the segmenting into memory, storing the data characterizing the segmenting in physical persistence, or transmitting the data characterizing the segmenting to a remote computing device over a network.
11. A system comprising:
at least one data processor; and
memory storing instructions which, when executed by the at least one data processor, result in operations comprising:
receiving data encapsulating a document of text;
segmenting the text into a plurality of semantically coherent units using a coherence-aware text segmentation (CATS) machine learning model, the CATS machine learning model being a multi-task learning model that alternatively minimizes a sentence-level segmentation objective and a coherence objective and which differentiates correct sequences of sentences in the document from corrupt sequences of sentences in the document;
providing data characterizing the segmenting;
wherein:
the corrupt sequences of sentences in the document are generated by: randomly shuffling an order of sentences in the document followed by randomly replacing sentences in the document;
the CATS model is generated by through cross-lingual zero-shot transfer in which a supervised alignment model is used to project target-language vectors from an independently trained embedding space of a target language to a monolingual embedding space of a source language, the supervised alignment model being based on solving a Procrustes problem.
12. The system of claim 11 , wherein the CATS machine learning model comprises two hierarchically connected transformer networks.
13. The system of claim 12 , wherein the document of text comprises a sequence of sentences of arbitrary length.
14. The system of claim 12 , wherein the operations further comprise:
encoding the text into a plurality of tokens prior to the segmenting.
15. The system of claim 14 , wherein the encoding comprises a concatenation of a pretrained word embedding and a position embedding.
16. The system of claim 14 , wherein a first of the transformer networks is a token-level transformer that encodes sentences from their corresponding tokens.
17. The system of claim 14 , wherein a second of the transformer networks is a sentence-level transformer which receives an output of the first transformer network and generates transformed sentence representations including a representation of a whole sequence of sentences.
18. The system of claim 17 , wherein the operations further comprise:
providing the transformed sentence representation to a feed forward segmentation classifier to generate a binary segmentation prediction for each sentence; and
feeding an encoding of the document of text to a neural network to generate a coherence score.
19. The system of claim 11 , wherein providing the data comprises at least one of: causing the data characterizing the segmenting to be displayed in a graphical user interface, loading the data characterizing the segmenting into memory, storing the data characterizing the segmenting in physical persistence, or transmitting the data characterizing the segmenting to a remote computing device over a network.
20. A computer-implemented method comprising:
receiving data encapsulating a document of text;
segmenting the text into a plurality of semantically coherent units using a coherence-aware text segmentation (CATS) machine learning model, the CATS machine learning model being a multi-task learning model that alternatively minimizes a sentence-level segmentation objective and a coherence objective and which differentiates correct sequences of sentences in the document from corrupt sequences of sentences in the document;
providing data characterizing the segmenting;
wherein:
the corrupt sequences of sentences in the document are generated by: receiving a training data set comprising a plurality of training documents each having a plurality of snippets having a plurality of sentences; for each snippet, generating a corrupt counterpart by randomly shuffling an order of sentences forming such snippet, randomly selecting a subset of the randomly shuffled sentences, and replacing the randomly selected sentences with randomly chosen sentences from other non-overlapping snippets;
the CATS model is generated by cross-lingual zero-shot transfer in which a supervised alignment model is used to project target-language vectors from an independently trained embedding space of a target language to a monolingual embedding space of a source language, the supervised alignment model being based on solving a Procrustes problem.