IP Library › Granted Patent US 12,664,974
Granted Patent B2
US 12,664,974 · App. 17/859,957 · Granted Jun 23, 2026

Training speech processing models using pseudo tokens

Inventors: Felix Wu (Ithaca, NY); Kwangyoun Kim (Santa Clara, CA); Ryan Thomas McDonald (London, GB); Kilian Quirin Weinberger (Ithaca, NY); Kyu Jeong Han (Pleasanton, CA); Yoav Artzi (New York, NY)
Assignee: ASAPP, INC.
G10L15/063G06N3/045G10L15/16G10L15/22G10L15/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,664,974
App. No.
17/859,957
Filed
Jul 7, 2022
Granted
Jun 23, 2026
Kind
B2
Art Unit
2658
USPC
704/232
Abstract

A speech processing model may be trained using pseudo tokens. Training a speech processing model with pseudo tokens may allow for training with a smaller amount of labeled training data and accordingly lower costs. A set of pseudo tokens may be determined by computing feature vectors from unlabeled training data, clustering the feature vectors, and performing token compression using the clustered feature vectors. A first speech processing model may be trained using unlabeled training data by determining sequences of pseudo tokens corresponding to the unlabeled training data. A second speech processing model may be initialized using the first speech processing model and then trained using labeled training data. The second speech processing model may then be deployed to a speech processing application.

Claims (58)

1 . A computer-implemented method, comprising:

obtaining a first training corpus of first training samples, wherein the first training samples are unlabeled;

computing feature vectors from the first training samples;

determining a set of pseudo tokens from the feature vectors by:

clustering at least a portion of the feature vectors into a plurality of clusters,

assigning a cluster-token to each cluster of the plurality of clusters,

determining a plurality of cluster-token sequences using the plurality of clusters, and

determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;

determining a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;

training a first neural network using the first training samples and the plurality of pseudo-token sequences;

obtaining a second training corpus of second training samples, wherein the second training corpus is labeled;

training a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and

deploying the second neural network to a production system.

2 . The computer-implemented method of claim 1 , wherein clustering the feature vectors comprises using k-means clustering.

3 . The computer-implemented method of claim 1 , comprising performing a deduplication operation on the plurality of cluster-token sequences.

4 . The computer-implemented method of claim 1 , wherein performing token compression comprises performing token compression using byte-pair encoding.

5 . The computer-implemented method of claim 1 , wherein performing token compression comprises performing token compression using subword regularization.

6 . The computer-implemented method of claim 1 , wherein the first neural network comprises a first encoder and a first decoder.

7 . The computer-implemented method of claim 1 , wherein the first neural network comprises a transducer neural network.

8 . The computer-implemented method of claim 7 , wherein the transducer neural network comprises a joint network.

9 . A system, comprising at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to:

obtain a first training corpus of first training samples, wherein the first training samples are unlabeled;

compute feature vectors from the first training samples;

determine a set of pseudo tokens from the feature vectors by:

clustering at least a portion of the feature vectors into a plurality of clusters,

assigning a cluster-token to each cluster of the plurality of clusters,

determining a plurality of cluster-token sequences using the plurality of clusters, and

determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;

determine a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;

train a first neural network using the first training samples and the plurality of pseudo-token sequences;

obtain a second training corpus of second training samples, wherein the second training corpus is labeled;

train a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and

deploy the second neural network to a production system.

10 . The system of claim 9 , wherein the first neural network comprises a first encoder and a first decoder.

11 . The system of claim 10 , wherein:

a second encoder of the second neural network is initialized using the first encoder of the first neural network; and

a second decoder of the second neural network is initialized using the first decoder of the first neural network.

12 . The system of claim 10 , wherein the first encoder comprises a transformer encoder and the first decoder comprises a transformer decoder.

13 . The system of claim 10 , wherein the first encoder comprises a recurrent neural network.

14 . The system of claim 10 , wherein the first encoder comprises a convolutional neural network.

15 . The system of claim 9 , wherein the at least one server computer is configured to compute the feature vectors by performing average pooling.

16 . One or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising:

obtaining a first training corpus of first training samples, wherein the first training samples are unlabeled;

computing feature vectors from the first training samples;

determining a set of pseudo tokens from the feature vectors by:

clustering at least a portion of the feature vectors into a plurality of clusters,

assigning a cluster-token to each cluster of the plurality of clusters,

determining a plurality of cluster-token sequences using the plurality of clusters, and

determining the set of pseudo tokens by performing token compression on the plurality of cluster-token sequences;

determining a plurality of pseudo-token sequences using the feature vectors and the set of pseudo tokens;

training a first neural network using the first training samples and the plurality of pseudo-token sequences;

obtaining a second training corpus of second training samples, wherein the second training corpus is labeled;

training a second neural network using the second training corpus, wherein at least a portion of the second neural network is initialized using the first neural network; and

deploying the second neural network to a production system.

17 . The one or more non-transitory, computer-readable media of claim 16 , wherein the first neural network comprises a first encoder and a first decoder.

18 . The one or more non-transitory, computer-readable media of claim 16 , wherein the second training corpus is labeled with text corresponding to the second training samples and the production system performs speech recognition.

19 . The one or more non-transitory, computer-readable media of claim 16 , wherein training the first neural network comprises optimizing a connectionist temporal classification objective function.

20 . The one or more non-transitory, computer-readable media of claim 16 , wherein a portion of the second neural network is initialized with random values.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 24, 2022
From: WU, FELIX; KIM, KWANGYOUN; MCDONALD, RYAN THOMAS; WEINBERGER, KILIAN QUIRIN; HAN, KYU JEONG; ARTZI, YOAV
To: ASAPP, INC.
Reel/Frame 060887/0562 →
Continuity (2)
Provisional Application 63303821 · Jan 27, 2022
Related Publication 20230237990A1 · Jul 27, 2023
References Cited (82)
US 8447604B1 · Chang · 2013 [cited by examiner]
US 20170372201A1 · Gupta et al. · 2017 [cited by applicant]
US 20190180175A1 · Meteer · 2019 [cited by examiner]
US 20200175961A1 · Thomson · 2020 [cited by examiner]
US 20210110813A1 · Khoury · 2021 [cited by examiner]
US 20210157983A1 · Kalluri · 2021 [cited by examiner]
US 20220101112A1 · Brown · 2022 [cited by examiner]
US 20220129749A1 · Moritz · 2022 [cited by examiner]
US 20220310113A1 · Tzinis · 2022 [cited by examiner]
US 20220374766A1 · Thorsley · 2022 [cited by examiner]
US 20230040181A1 · Neelagiri · 2023 [cited by examiner]
US 20230078698A1 · Shnarch · 2023 [cited by examiner]
US 20230196710A1 · Pan · 2023 [cited by examiner]
US 20250061277A1 · Zhu · 2025 [cited by examiner]
Amodei, Dario et al., “Deep speech 2: End-to-end speech recognition in english and mandarin”, International conference on machine learning, PMLR, 2016, pp. 173-182. [cited by applicant]
Arthur, David et al., “k-means++: the advantages of careful seeding”, SODA '07: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, 2007, pp. 1027-1035. [cited by applicant]
Babu, Arun et al., “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”, arXiv:2111.09296v3 [cs.CL], https://arxiv.org/pdf/2111.09296 (accessed on Aug. 20, 2024), Dec. 16, 2021, 23 pages. [cited by applicant]
Baevski, Alexei et al., “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language”, arXiv:2202.03555v3 [cs.LG], https://arxiv.org/pdf/2202.03555 (accessed on Aug. 20, 2024), Oct. 25, 202… [cited by applicant]
Baevski, Alexei et al., “Effectiveness of self-supervised pre-training for speech recognition”, arXiv:1911.03912v3 [cs.CL], https://arxiv.org/pdf/1911.03912.pdf (accessed on Oct. 4, 2021), May 18, 2020, 8 pages. [cited by applicant]
Baevski, Alexei et al., “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”, arXiv:2006.11477v3 [cs.CL], https://arxiv.org/pdf/2006.11477.pdf (accessed on Oct. 4, 2021), Oct. 22, 2020, 19 p… [cited by applicant]
Bahdanau, Dzmitry et al., “Neural Machine Translation by Jointly Learning to Align and Translate”, CoRR abs/1409.0473, http://arxiv.org/abs/1409.0473, May 19, 2016, 15 pages. [cited by applicant]
Caron, Mathilde et al., “Emerging Properties in Self-Supervised Vision Transformers”, arXiv:2104.14294v2 [cs.CV], https://arxiv.org/pdf/2104.14294 (accessed on Aug. 20, 2024), May 24, 2021, 21 pages. [cited by applicant]
Chan, William et al., “Listen, Attend and Spell: A neural network for large vocabulary conversational speech recognition”, International Conference on Acoustics, Speech, and Signal Processing, pp. 4960-4964, 2016, arXiv… [cited by applicant]
Chen, Sanyuan et al., “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing”, arXiv:2110.13900v5 [cs.CL], https://arxiv.org/pdf/2110.13900 (accessed on Aug. 20, 2024), Jun. 17, 2022, 14 pages. [cited by applicant]
Chen, Ting et al., “A simple framework for contrastive learning of visual representations”, Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020, ArXiv, abs/2002.05709v3 … [cited by applicant]
Child, Rewon et al., “Generating Long Sequences with Sparse Transformers”, arXiv:1904.10509v1 [cs.LG], https://arxiv.org/pdf/1904.10509 (accessed on Aug. 20, 2024), Apr. 23, 2019, 10 pages. [cited by applicant]
Chung, Yu-An et al., “W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training”, arXiv:2108.06209v2 [cs.LG], https://arxiv.org/pdf/2108.06209 (accessed on Aug. 20, 20… [cited by applicant]
Collobert, Ronan et al., “Wav2letter: an end-to-end convnet-based speech recognition system”, ArXiv, abs/1609.03193v2 [cs.LG], Sep. 13, 2016, 8 pages. [cited by applicant]
Conneau, Alexis et al., “Unsupervised cross-lingual representation learning for speech recognition”, ArXiv, abs/2006.13979v2 [cs.CL], Dec. 15, 2020, 12 pages. [cited by applicant]
Dai, Zihang et al., “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, arXiv:1901.02860v3 [cs.LG], https://arxiv.org/pdf/1901.02860.pdf, Jun. 2, 2019, 20 pages. [cited by applicant]
Devlin, J. et al., “Bert: Pre-training of deep bidirectional transformers for language understanding”, Proceedings of NAACL-HLT 2019, Minneapolis, Minnesota, Jun. 2-7, 2019, pp. 4171-4186. [cited by applicant]
Gage, Philip , “A New Algorithm for Data Compression”, C Users J. 12(2):23-38, http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045.HTM (accessed on Aug. 19, 2024), Feb. 1994, 11 pages. [cited by applicant]
Ghannay, Sahar et al., “End-to-end named entity and semantic concept extraction from speech”, IEEE Spoken Language Technology Workshop, Athens, Greece. hal-01987740v2, https://hal.science/hal-01987740v2/document, Dec. 2… [cited by applicant]
Graves, Alex et al., “Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks”, International Conference on Machine Learning, 2006, pp. 369-376. [cited by applicant]
Graves, Alex , “Sequence Transduction with Recurrent Neural Networks”, arXiv:1211.3711v1 [cs.NE], https://arxiv.org/pdf/1211.3711.pdf, Nov. 14, 2012, 9 pages. [cited by applicant]
Grill, Jean-Bastien et al., “Bootstrap your own latent: A new approach to self-supervised learning”, arXiv preprint arXiv:2006.07733v3 [cs.LG], Sep. 10, 2020, 35 pages. [cited by applicant]
Gulati, Anmol et al., “Conformer: Convolution-augmented Transformer for Speech Recognition”, arXiv: 2005.08100v1 [eess.AS], https://arxiv.org/pdf/2005.08100.pdf, May 16, 2020, 5 pages. [cited by applicant]
Han, Wei et al., “ContextNet: Improving convolutional neural networks for automatic speech recognition with global context”, arXiv:2005.03191v3 [eess.AS], https://arxiv.org/pdf/2005.03191.pdf, May 16, 2020, 5 pages. [cited by applicant]
He, Kaiming et al., “Momentum Contrast for Unsupervised Visual Representation Learning”, CVPR, https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_… [cited by applicant]
Hsu, Wei-Ning et al., “Hubert: How much can a bad teacher benefit asr pre-training”, Neural Information Processing Systems Workshop on Self-Supervised Learning for Speech and Audio Processing Workshop / arXiv:2106.07447… [cited by applicant]
Hsu, Wei-Ning et al., “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”, IEEE/ACM Trans. Audio, Speech and Lang. Proc., vol. 29, 2021, pp. 3451-3460. [cited by applicant]
Hsu, Wei-Ning et al., “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training”, ArXiv, abs/2104.01027v2 [cs.SD], Sep. 8, 2021, 9 pages. [cited by applicant]
Inaguma, Hirofumi et al., “ESPnet-ST: All-in-One Speech Translation Toolkit”, arXiv:2004.10234v2 [cs.CL], https://arxiv.org/pdf/2004.10234 (accessed on Aug. 20, 2024), Sep. 30, 2020, 10 pages. [cited by applicant]
Kahn, J. et al., “Libri-Light: A benchmark for ASR with limited or no supervision”, [Online]. Available: http://arxiv.org/abs/1912.07875, 2019, 8 pages. [cited by applicant]
Kahn, Jacob et al., “Librilight: A benchmark for asr with limited or no supervision”, ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), https://github.com/facebookresear… [cited by applicant]
Kudo, Taku et al., “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing”, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing … [cited by applicant]
Kudo, Taku , “Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates”, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long … [cited by applicant]
Lai, Cheng-I J. et al., “PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition”, arXiv:2106.05933v2 [cs.CL], https://arxiv.org/pdf/2106.05933 (accessed on Aug. 20, 2024), Oct. 26, 2021, 53 pages. [cited by applicant]
Lakhotia, Kushal et al., “On Generative Spoken Language Modeling from Raw Audio”, arXiv:2102.01192v2 [cs.CL], https://arxiv.org/pdf/2102.01192 (accessed on Aug. 20, 2024), Sep. 9, 2021, 20 pages. [cited by applicant]
Lewis, Mike et al., “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension”, Proceedings of the 58th Annual Meeting of the Association for Computational Lingui… [cited by applicant]
Li, Chunyuan et al., “Efficient Self-supervised Vision Transformers for Representation Learning”, arXiv:2106.09785v2 [cs.CV], https://arxiv.org/pdf/2106.09785 (accessed Aug. 21, 2024), Jul. 6, 2022, 27 pages. [cited by applicant]
Li, Xian et al., “Multilingual Speech Translation with Efficient Finetuning of Pretrained Models”, arXiv:2010.12829v4 [cs.CL], https://arxiv.org/pdf/2010.12829 (accessed on Aug. 21, 2024), Jan. 2, 2021, 12 pages. [cited by applicant]
Liu, Yinhan et al., “Multilingual Denoising Pre-training for Neural Machine Translation”, Transactions of the Association for Computational Linguistics, vol. 8, 2020, pp. 726-742. [cited by applicant]
Liu, Yinhan et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, arXiv:1907.11692v1 [cs.CL], Jul. 26, 2019, 13 Pages. [cited by applicant]
Mohamed, Abdelrahman et al., “Transformers with convolutional context for ASR”, arXiv:1904.11660v2 [cs.CL], https://arxiv.org/pdf/1904.11660 (accessed on Aug. 21, 2024), Mar. 2, 2020, 5 pages. [cited by applicant]
Ott, Myle et al., “FAIRSEQ: A Fast, Extensible Toolkit for Sequence Modeling”, Proceedings of NAACL-HLT 2019: Demonstration, arXiv:1904.01038v1 [cs.CL], http://128.84.21.203/pdf/1904.01038, Apr. 1, 2019, 6 pages. [cited by applicant]
Panayotov, Vassil et al., “Librispeech: An ASR Corpus Based On Public Domain Audio Books”, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206-5210. [cited by applicant]
Pasad, Ankita et al., “On the Use of External Data for Spoken Named Entity Recognition”, arXiv:2112.07648v2 [cs.CL], https://arxiv.org/pdf/2112.07648 (accessed on Aug. 21, 2024), Jul. 9, 2022, 14 pages. [cited by applicant]
Paszke, Adam et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library”, Advances in Neural Information Processing Systems 32, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019),… [cited by applicant]
Pedregosa, Fabian et al., “Scikit-learn: Machine Learning in Python”, Journal of Machine Learning Research, vol. 12, 2011, pp. 2825-2830. [cited by applicant]
Peters, Matthew E. et al., “Deep contextualized word representations”, Proceedings of NAACL-HLT 2018, New Orleans, Louisiana, Jun. 1-6, 2018, pp. 2227-2237. [cited by applicant]
Pratap, Vineel et al., “Wav2letter++: A fast open-source speech recognition system”, ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6460-6464, 2019, arXiv:1812.07… [cited by applicant]
Qi, Weizhen et al., “ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training”, rXiv:2001.04063v3 [cs.CL], https://arxiv.org/pdf/2001.04063 (accessed on Aug. 21, 2024), Oct. 21, 2020, 10 pages. [cited by applicant]
Radford, Alec et al., “Language Models are Unsupervised Multitask Learners”, Technical report, OpenAi, https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf (accessed on Aug. 21, 2024), Feb. 14, 2019, 24 p… [cited by applicant]
Raffel, Colin et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, Journal of Machine Learning Research 21 (2020) pp. 1-67, arXiv:1910.10683v4 [cs.LG] https://arxiv.org/pdf/1910.1… [cited by applicant]
Sanh, Victor et al., “Multitask Prompted Training Enables Zero-Shot Task Generalization”, Published as a conference paper at ICLR 2022, arXiv:2110.08207v3 [cs.LG], https://arxiv.org/pdf/2110.08207 (accessed on Aug. 21, … [cited by applicant]
Schneider, Steffen et al., “wav2vec: Unsupervised pre-training for speech recognition”, Interspeech, arXiv:1904.05862v4 [cs.CL], Sep. 11, 2019, 9 pages. [cited by applicant]
Sculley, D. , “Web-Scale K-Means Clustering”, WWW 2010, Raleigh, NC, USA, https://citeseerx.ist.psu.edu/document?repid=rep1type=pdf&doi=b452a856a3e3d4d37b1de837996aa6813bedfdcf (accessed on Aug. 21, 2024), Apr. 26-30, 2… [cited by applicant]
Shon, Suwon et al., “SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech”, arXiv:2111.10367v3 [cs.CL], https://arxiv.org/pdf/2111.10367 (accessed on Aug. 21, 2024), Jul. 29, 2022, 8 … [cited by applicant]
Sutskever, Ilya et al., “Sequence to Sequence Learning with Neural Networks”, CoRR abs/1409.3215, http://arxiv.org/abs/1409.3215, 2014, 9 pages. [cited by applicant]
Van Den Oord, Aaron et al., “Representation learning with contrastive predictive coding”, arXiv:1807.03748v2 [cs.LG], Jan. 22, 2019, 13 pages. [cited by applicant]
Wang, Changhan et al., “CoVoST 2 and Massively Multilingual Speech-to-Text Translation”, arXiv:2007.10310v3 [cs.CL], https://arxiv.org/pdf/2007.10310 (accessed on Aug. 21, 2024), Oct. 24, 2020, 6 pages. [cited by applicant]
Wang, Changhan et al., “FAIRSEQ S2T: Fast Speech-to-Text Modeling with FAIRSEQ”, arXiv:2010.05171v2 [cs.CL], https://arxiv.org/pdf/2010.05171 (accessed on Aug. 21, 2024), Jun. 14, 2022, 8 pages. [cited by applicant]
Wang, Changhan et al., “Large-scale self and semi-supervised learning for speech translation”, arXiv:2104.06678v1 [cs.CL], Apr. 14, 2021, 5 pages. [cited by applicant]
Wang, Changhan et al., “Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation”, arXiv:2101.00390v2 [cs.CL], Jul. 27, 2021, 11 pages. [cited by applicant]
Watanabe, Shinji et al., “ESPnet: End-to-End Speech Processing Toolkit”, arXiv:1804.00015v1 [cs.CL], https://arxiv.org/pdf/1804.00015.pdf, Mar. 30, 2018, 5 pages. [cited by applicant]
Wu, Felix et al., “Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition”, arXiv:2109.06870v1 [cs.CL], https://arxiv.org/pdf/2109.06870 (accessed on Aug. 21, 2024), Sep. 14, 2021, 19 page… [cited by applicant]
Wu, Zhirong et al., “Unsupervised feature learning via non-parametric instance discrimination”, CVPR, 2018, arXiv:1805.01978v1 [cs.CV], https://arxiv.org/pdf/1805.01978, May 5, 2018, 10 pages. [cited by applicant]
Xu, Qiantong et al., “Self-training and pre-training are complementary for speech recognition”, arXiv:2010.11430v1 [cs.LG], Oct. 22, 2020, 8 pages. [cited by applicant]
Yadav, Hemant et al., “End-to-end Named Entity Recognition from English Speech”, Interspeech 2020, arXiv:2005.11184v1 [cs.CL], https://arxiv.org/pdf/2005.11184 (accessed on Aug. 21, 2024), May 22, 2020, 5 pages. [cited by applicant]
Yang, Shu-Wen et al., “SUPERB: Speech processing Universal PERformance Benchmark”, arXiv:2105.01051v4 [cs.CL], https://arxiv.org/pdf/2105.01051 (accessed on Aug. 21, 2024), Oct. 15, 2021, 6 pages. [cited by applicant]
Zhang, Jingqing et al., “PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization”, Proceedings of the 37th International Conference on Machine Learning, Online, PMLR 119, 2020, pp. 11328-11339. [cited by applicant]