System and method for generating thoughts with large language models using codewords
This invention presents an optimized approach for training and operating Large Language Models (LLMs) using codewords. By converting traditional token-based LLMs to codeword-based systems, the method achieves significant efficiency gains. The process involves tokenizing training data and assigning codewords to tokens. LLMs are then trained and operated using these compact codewords instead of conventional tokens. During operation, prompts are converted to codewords, processed by the LLM, and the outputs are converted back to text. This approach reduces the overall cost of training and operating LLMs by approximately, offering a more efficient solution for large-scale language processing tasks.
1 . A computer system configured to execute software instructions stored on nontransitory machine-readable storage media, wherein the software instructions comprise instructions that:
tokenize a set of training data into a plurality of training tokens;
create a codeword dictionary by assigning unique codewords to each of the plurality of training tokens;
convert all training tokens into a plurality of training codewords using the codeword dictionary;
train a large codeword model using the plurality of training codewords;
receive a plurality of prompt codewords;
process the plurality of prompt codewords through the large codeword model to generate a plurality of thought codewords representing intermediate reasoning steps; and
process the sequence of prompt codewords together with the plurality of thought codewords through the large codeword model to generate a codeword response.
2 . The computer system of claim 1 , wherein the large codeword model uses a latent transformer architecture.
3 . The computer system of claim 1 , wherein conversion between tokens and codewords for a user prompt is performed on an edge device using a local codeword dictionary lookup, and only codewords are transmitted to the large codeword model.
4 . The computer system of claim 1 , further comprising a plurality of language-specific codebooks and a codeword translator configured to map codewords between different language codebooks.
5 . The computer system of claim 1 , wherein the large codeword model includes a codeword clustering module that groups semantically related codewords and learns a single embedding per cluster.
6 . The computer system of claim 1 , wherein the model comprises multiple embedding layers respectively configured for different input modalities and a concatenation stage that forms a unified sequence for transformer processing.
7 . The computer system of claim 1 , further comprising a thought association layer configured to (i) segment the prompt into semantic portions, (ii) associate each thought codeword with at least one segment, and (iii) compute a relevance score for each association.
8 . The computer system of claim 1 , wherein a thought cache stores thought codewords together with (a) a reasoning-chain index recording prerequisite, supporting, or contradictory relationships among thoughts, and (b) cross-references enabling retrieval of complete reasoning sequences.
9 . The computer system of claim 2 , wherein the latent transformer operates over cluster-level embeddings produced by the codeword clustering module.
10 . The computer system of claim 1 , wherein the intermediate reasoning steps are identified by scanning hidden states for activation transitions and projecting them via a learned matrix before codeword encoding.
11 . A computer-implemented method comprising the steps of:
tokenizing a set of training data into a plurality of training tokens;
creating a codeword dictionary by assigning unique codewords to each of the plurality of training tokens;
converting all training tokens into a plurality of training codewords using the codeword dictionary;
training a large codeword model using the plurality of training codewords;
receiving a plurality of prompt codewords;
processing the plurality of prompt codewords through the large codeword model to generate a plurality of thought codewords representing intermediate reasoning steps; and
processing the sequence of prompt codewords together with the plurality of thought codewords through the large codeword model to generate a codeword response.
12 . The computer-implemented method of claim 11 , wherein the large codeword model uses a latent transformer architecture.
13 . The computer-implemented method of claim 11 , wherein conversion between tokens and codewords for a user prompt is performed on an edge device using a local codeword dictionary lookup, and only codewords are transmitted to the large codeword model.
14 . The computer-implemented method of claim 11 , further comprising a plurality of language-specific codebooks and a codeword translator configured to map codewords between different language codebooks.
15 . The computer-implemented method of claim 11 , wherein the large codeword model includes a codeword clustering module that groups semantically related codewords and learns a single embedding per cluster.
16 . The computer-implemented method of claim 11 , wherein the model comprises multiple embedding layers respectively configured for different input modalities and a concatenation stage that forms a unified sequence for transformer processing.
17 . The computer-implemented method of claim 11 , further comprising a thought association layer configured to (i) segment the prompt into semantic portions, (ii) associate each thought codeword with at least one segment, and (iii) compute a relevance score for each association.
18 . The computer-implemented method of claim 11 , wherein a thought cache stores thought codewords together with (a) a reasoning-chain index recording prerequisite, supporting, or contradictory relationships among thoughts, and (b) cross-references enabling retrieval of complete reasoning sequences.
19 . The computer-implemented method of claim 12 , wherein the latent transformer operates over cluster-level embeddings produced by the codeword clustering module.
20 . The computer-implemented method of claim 11 , wherein the intermediate reasoning steps are identified by scanning hidden states for activation transitions and projecting them via a learned matrix before codeword encoding.