Large-scale text data encoding and compression
Embodiments of the present invention provide an approach for compressing data, and more particularly, to large-scale text data encoding and compression using absolute overfitting on pre-trained language models. Large-scale data is parsed into sentences. A unique token is generated for each sentence to form a token list. A generative (or compression) model is trained from the tokens in the token list to produce the corresponding sentence of each token through absolute overfitting of a pre-trained language model. The compressed text data is stored as the token list and generative model, resulting in a storage space savings.
1. A computer-implemented method for encoding and compressing text data, comprising computer-implemented steps of:
parsing, using a pre-trained language model, received text data to be compressed into a set of sentences;
generating, using the pre-trained language model, a unique token for each sentence among the set of sentences to form a token list;
training a generative model using absolute overfitting on the pre-trained language model, wherein the training further comprises training a decoder part of the generative model such that an output generated for a given input that includes the unique token matches an original sentence in the received text data corresponding to the unique token; and
compressing the received text data using the trained generative model and each unique token within the token list, wherein the compressing further comprises using the trained generative model to produce a corresponding sentence of each unique token through the absolute overfitting of the pre-trained language model, wherein a combination of the token list and the trained generative model that is based on the absolute overfitting of the pre-trained language model represent the compressed text data.
2. The computer-implemented method of claim 1 , wherein the unique token includes an embedding, a length of the corresponding sentence, and a hash identifier.
3. The computer-implemented method of claim 1 , wherein the pre-trained language model is a Generative Pre-trained Transformer 3 (GPT-3) or Bidirectional Encoder Representations from Transformers (BERT) model.
4. The computer-implemented method of claim 1 , wherein training the generative model further comprises using the unique token within the token list as input to produce the output and ensuring, using a length of the corresponding sentence stored in the unique token as a constraint, the output exactly matches a text of the corresponding sentence.
5. The computer-implemented method of claim 1 , further comprising decompressing each sentence in the text data using the trained generative model and the token list.
6. The computer-implemented method of claim 5 , further comprising organizing, using a hash identifier of each unique token in the token list, the decompressed sentences in an exact order of the received text data, wherein the hash identifier of each unique token references a text unit of the corresponding sentence.
7. The computer-implemented method of claim 2 , wherein the embedding of the unique token includes a sentence vector of the sentence corresponding to the token.
8. A system for encoding and compressing text data, comprising:
a memory medium comprising program instructions;
a bus coupled to the memory medium; and
a processor, for executing the program instructions, coupled to the memory medium that when executing the program instructions causes the system to:
parse, using a pre-trained language model, received text data to be compressed into a set of sentences;
generate, using the pre-trained language model, a unique token for each sentence among the set of sentences to form a token list;
train a generative model using absolute overfitting on the pre-trained language model, wherein the training further comprises training a decoder part of the generative model such that an output generated for a given input that includes the unique token matches an original sentence in the received text data corresponding to the unique token; and
compress the received text data using the trained generative model and each unique token within the token list, wherein the compressing further comprises using the trained generative model to produce a corresponding sentence of each unique token through the absolute overfitting of the pre-trained language model, wherein a combination of the token list and the trained generative model that is based on the absolute overfitting of the pre-trained language model represent the compressed text data.
9. The system of claim 8 , wherein the unique token includes an embedding, a length of the corresponding sentence, and a hash identifier.
10. The system of claim 8 , wherein the pre-trained language model is a Generative Pre-trained Transformer 3 (GPT-3) or Bidirectional Encoder Representations from Transformers (BERT) model.
11. The system of claim 8 , the memory medium further comprising instructions to train the generative model using the unique token within the token list as input to produce the output and ensuring, using a length of the corresponding sentence stored in the unique token as a constraint, the output exactly matches a text of the corresponding sentence.
12. The system of claim 8 , the memory medium further comprising instructions to decompress each sentence in the text data using the trained generative model and the token list.
13. The system of claim 12 , the memory medium further comprising instructions to organize, using a hash identifier of each unique token in the token list, the decompressed sentences in an exact order of the received text data, wherein the hash identifier of each unique token references a text unit of the corresponding sentence.
14. The system of claim 9 , wherein the embedding of the unique token includes a sentence vector of the sentence corresponding to the unique token.
15. A computer program product for encoding and compressing text data, the computer program product comprising a computer readable storage device, and program instructions stored on the computer readable storage device, to:
parse, using a pre-trained language model, received text data to be compressed into a set of sentences;
generate, using the pre-trained language model, a unique token for each sentence among the set of sentences to form a token list;
train a generative model using absolute overfitting on the pre-trained language model, wherein the training further comprises training a decoder part of the generative model such that an output generated for a given input that includes the unique token matches an original sentence in the received text data corresponding to the unique token; and
compress the received text data using the trained generative model and each unique token within the token list, wherein the compressing further comprises using the trained generative model to produce a corresponding sentence of each unique token through the absolute overfitting of the pre-trained language model, wherein a combination of the token list and the trained generative model that is based on the absolute overfitting of the pre-trained language model represent the compressed text data.
16. The computer program product of claim 15 , wherein the unique token includes an embedding, a length of the corresponding sentence, and a hash identifier.
17. The computer program product of claim 15 , wherein the pre-trained language model is a Generative Pre-trained Transformer 3 (GPT-3) or Bidirectional Encoder Representations from Transformers (BERT) model.
18. The computer program product of claim 15 , further comprising program instructions stored on the computer readable storage device to train the generative model using the unique token within the token list as input to produce the output and ensuring, using a length of the corresponding sentence stored in the unique token as a constraint, the output exactly matches a text of the corresponding sentence.
19. The computer program product of claim 15 , further comprising program instructions stored on the computer readable storage device to decompress each sentence in the text data using the trained generative model and the token list.
20. The computer program product of claim 19 , further comprising program instructions stored on the computer readable storage device to organize, using a hash identifier of each unique token in the token list, the decompressed sentences in an exact order of the received text data, wherein the hash identifier of each unique token references a text unit of the corresponding sentence.