Training and domain adaptation for supervised text segmentation
Data such as unstructured text is received that includes a sequence of sentences. This received data is then tokenized into a plurality of tokens. The received data is segmented using a hierarchical transformer network model including a token transformer, a sentence transformer, and a segmentation classifier. The token transformer contextualizes tokens within sentences and yields sentence embeddings. The sentences transformer contextualizes sentence representations based on the sentence embedddings. The segmentation classifier predicts segments of the received data based on the contextualized sentence representations. Data can be provided which characterizes the segmentation of the received data. Related apparatus, systems, techniques and articles are also described.
1. A computer-implemented method for text segmentation, comprising:
receiving an unstructured dataset built from a plurality of educational reading materials, wherein the unstructured dataset comprises a sequence of sentences;
tokenizing the unstructured dataset into a plurality of tokens;
contextualizing the plurality of tokens within the sequence of sentences by a token transformer and yielding sentence embeddings, wherein at least the token transformer was previously fine-tuned on a structured dataset not built from the plurality of educational reading materials;
contextualizing sentence representations by a sentence transformer based on the sentence embeddings, wherein the sentence transformer and the token transformer are hierarchically linked;
predicting at least one segment of the unstructured dataset by a segmentation classifier based on the contextualized sentence representations; and
providing data characterizing the at least one predicted segment of the unstructured dataset.
2. The method of claim 1 , wherein the providing data comprises at least one of: causing the provided data to be displayed in an electronic visual display, transmitting the provided data to a remote computing device, storing the provided data in physical persistence, or loading the provided data into memory.
3. The method of claim 1 , wherein the token transformer comprises a plurality of layers, wherein at least one layer of the plurality comprises a multi-head attention sublayer and a feed-forward sublayer.
4. The method of claim 3 , wherein the at least one layer of the plurality comprises a first bottleneck adapter positioned after the multi-head attention sublayer and a second bottleneck adapter positioned after the feed-forward sublayer.
5. The method of claim 4 , wherein the token transformer comprises a plurality of original parameters that are initialized with weights of a pretrained RoBERTa model.
6. The method of claim 5 , wherein the pretrained RoBERTa weights encode general-purpose distributional knowledge.
7. The method of claim 6 , wherein the plurality of original parameters remains unchanged throughout adapter-based fine-tuning, such that the distributional knowledge is preserved.
8. The method of claim 7 , wherein the at least one layer of the plurality is augmented with adapter parameters that are updated during adapter-based fine-tuning.
9. The method of claim 8 , wherein both the plurality of original parameters and the adapter parameters are configured to remain unchanged after the unstructured dataset is received.
10. The method of claim 1 , wherein the structured dataset comprises a segment-annotated sequence of sentences.
11. The method of claim 10 , wherein the annotated dataset is segmented by a heading structure.
12. The method of claim 1 , wherein the sentence transformer is randomly initiated.
13. The method of claim 1 , wherein the unstructured dataset is synthetically created at least by selecting and concatenating two samples from a same book, different books from a same content area, different books from different content areas, or a combination thereof.
14. The method of claim 1 , further comprising sliding a window over at least one sentence of the sequence to generate a plurality of probabilities that the at least one sentence starts a new segment.
15. The method of claim 14 , wherein the plurality of probabilities is averaged, and the segmentation classifier is configured to predict that the at least one sentence starts a new segment if the average is above a predetermined threshold.
16. A system for text segmentation, comprising:
at least one data processor; and
a memory including instructions which, when executed by the at least one data processor, result in operations comprising:
receiving an unstructured dataset built from a plurality of educational reading materials, wherein the unstructured dataset comprises a sequence of sentences;
tokenizing the unstructured dataset into a plurality of tokens;
contextualizing the plurality of tokens within the sequence of sentences by a token transformer and yielding sentence embeddings, wherein at least the token transformer was previously fine-tuned on a structured dataset not built from the plurality of educational reading materials;
contextualizing sentence representations by a sentence transformer based on the sentence embeddings, wherein the sentence transformer and the token transformer are hierarchically linked;
predicting at least one segment of the unstructured dataset by a segmentation classifier based on the contextualized sentence representations; and
providing data characterizing the at least one segment of the unstructured dataset.
17. A computer-implemented method of training a neural network for text segmentation, comprising:
receiving a first training dataset from a non-educational domain, the first dataset comprising a sequence of sentences;
training a hierarchical transformer network model out-of-domain, comprising:
tokenizing the first dataset into a plurality of tokens;
initializing weights of parameters of a token transformer;
contextualizing the plurality of tokens within the sequence of sentences by the token transformer and yielding a plurality of sentence embeddings;
contextualizing a plurality of sentence representations by a sentence transformer based on the plurality of sentence embeddings;
predicting, by a segmentation classifier, whether a sentence in the sequence starts a segment based on the plurality of contextualized sentence representations;
updating at least one adapter parameter of the token transformer;
receiving a second training dataset from an educational domain, the second dataset comprising a second sequence of sentences;
training the hierarchical transformer network model in-domain, comprising:
tokenizing the second dataset into a second plurality of tokens;
initializing the token transformer with the at least one updated adapter parameter;
contextualizing the second plurality of tokens within the second sequence of sentences by the token transformer and yielding a second plurality of sentence embeddings;
contextualizing a second plurality of sentence representations by the sentence transformer based on the second plurality of sentence embeddings; and
predicting, by the segmentation classifier, whether a second sentence in the second sequence starts a second segment based on the second plurality of contextualized sentence representations.
18. The method of claim 17 , wherein the weights of parameters of the token transformer comprise of a plurality of RoBERTa parameters.
19. The method of claim 18 , wherein the plurality of RoBERTa parameters are frozen throughout the in-domain training.
20. The method of claim 17 , wherein the educational domain is designed for grade-1 to college-level student populations.