System and method for generating thoughts with large language models using codewords
This invention presents an optimized approach for training and operating Large Language Models (LLMs) using codewords. By converting traditional token-based LLMs to codeword-based systems, the method achieves significant efficiency gains. The process involves tokenizing training data and assigning codewords to tokens. LLMs are then trained and operated using these compact codewords instead of conventional tokens. During operation, prompts are converted to codewords, processed by the LLM, and the outputs are converted back to text. This approach reduces the overall cost of training and operating LLMs by approximately, offering a more efficient solution for large-scale language processing tasks.
1 . A computer system comprising:
a hardware memory, wherein the computer system is configured to execute software instructions stored on nontransitory machine-readable storage media that:
tokenize a set of training data into a plurality of training tokens;
create a codeword dictionary by assigning unique codewords to each of the plurality of training tokens;
convert all training tokens into a plurality of training codewords using the codeword dictionary;
train a large language model using the plurality of training codewords;
receive a text prompt from a user;
tokenize the prompt into a plurality of prompt tokens;
convert the plurality of prompt tokens into a plurality of prompt codewords using the codeword dictionary;
process the plurality of prompt codewords through a large language model to generate a plurality of thought codewords representing intermediate reasoning steps;
associate each thought codeword with a corresponding portion of the prompt;
encode the thought codewords with metadata; and
store the thought codewords and their associated metadata.
2 . The system of claim 1 , wherein the text prompt is received, tokenized, and converted from tokens to codewords and from codewords back to tokens on an edge device.
3 . The system of claim 2 , wherein the codeword dictionary is a local codeword dictionary lookup on the edge device.
4 . The system of claim 1 , wherein the large language model uses a transformer architecture.
5 . The system of claim 1 , wherein the large language model uses a latent transformer architecture.
6 . The computer system of claim 1 , further configured to:
process the sequence of prompt codewords through the large language model to generate a codeword response; and
convert the codeword response into a text response.
7 . The computer system of claim 1 , further configured to:
generate a response based on both the prompt codewords and the generated thought codewords.
8 . A computer-implemented method comprising the steps of:
tokenizing a set of training data into a plurality of training tokens;
creating a codeword dictionary by assigning unique codewords to each of the plurality of training tokens;
converting all training tokens into a plurality of training codewords using the codeword dictionary;
training a large language model using the plurality of training codewords;
receiving a text prompt from a user;
tokenizing the prompt into a plurality of tokens;
converting the plurality of tokens into a plurality of prompt codewords using the codeword dictionary;
processing the plurality of prompt codewords through a large language model to generate a plurality of thought codewords representing intermediate reasoning steps;
associating each thought codeword with a corresponding portion of the prompt;
encoding the thought codewords with metadata; and
storing the thought codewords and their associated metadata.
9 . The method of claim 8 , wherein the text prompt is received, tokenized, and converted from tokens to codewords and from codewords back to tokens on an edge device.
10 . The method of claim 9 , wherein the codeword dictionary is a local codeword dictionary lookup on the edge device.
11 . The method of claim 8 , wherein the large language model uses a transformer based architecture.
12 . The method of claim 8 , wherein the large language model uses a variational autoencoder based architecture.
13 . The method of claim 8 , further comprising the steps of:
processing the sequence of prompt codewords through the large language model to generate a codeword response; and
convert the codeword response into a text response.
14 . The method of claim 8 , further comprising the steps of:
generating a response based on both the prompt codewords and the generated thought codewords.