Hierarchical smart caching for machine learning codeword responses
A system and method for deep learning using a large codeword model with hierarchical caching is disclosed. The system processes input prompts into tokens, maps them to codewords using a codebook, and processes these through a machine learning core to generate responses. A sophisticated caching architecture stores and retrieves responses across both local and global cache tiers. The local cache maintains frequently accessed responses on edge devices through short-term and persistent storage components, while the global cache enables knowledge sharing across multiple devices. A context aggregator identifies relationships between cached responses to form comprehensive contextual representations. This hierarchical caching system significantly reduces computational requirements by reusing previously generated responses for similar prompts, while continuously optimizing cache contents based on relevance scoring and usage patterns. The approach enables efficient scaling across distributed environments while maintaining response quality.
1 . A computer system comprising:
a hardware memory, wherein the computer system is configured to execute software instructions stored on nontransitory machine-readable storage media that:
receive a prompt from a user;
process the prompt into a plurality of tokens;
map the plurality of tokens to a plurality of codewords using a codebook, where each token is assigned a specific codeword;
train a machine learning core on a plurality of codewords;
process the plurality of codewords through the trained machine learning core to generate a codeword response;
store the codeword response and associated processing outputs from the machine learning core in a hierarchical caching system comprising:
a local cache configured to store the codeword responses on an edge device; and
a global cache configured to store the codeword responses across multiple edge devices;
evaluate relevance of the stored codeword responses;
retrieve relevant codeword responses when processing similar prompts; and
translate the codeword response into a response which matches a modality of the prompt.
2 . The system of claim 1 , wherein the machine learning core is a conventional transformer-based architecture comprising:
an embedding layer;
a positional encoding layer; and
a series of transformer layers.
3 . The system of claim 2 , wherein computer system is further configured to:
split the codewords into smaller units before processing through the conventional transformer-based architecture.
4 . The system of claim 1 , wherein the machine learning core is a latent transformer core comprising:
a variational autoencoder with an encoder and a decoder; and
a transformer that processes latent space vectors, wherein the transformer does not include an embedding layer and a positional encoding layer.
5 . The system of claim 4 , wherein processing the plurality of codewords through the machine learning core comprises:
generating a plurality of latent space vectors by processing the plurality of codewords through the variational autoencoder's encoder;
learning relationships between the plurality of latent space vectors by processing them through the transformer;
using the learned relationships to generate a plurality of output latent space vectors; and
generating the codeword response by passing the output latent space vectors through the variational autoencoder's decoder.
6 . The system of claim 1 , wherein computer system is further configured to:
split the latent space vectors into smaller units before processing through the transformer.
7 . A method for hierarchical smart caching for machine learning codeword responses, comprising the steps of:
receiving a prompt from a user;
processing the prompt into a plurality of tokens;
mapping the plurality of tokens to a plurality of codewords using a codebook, where each token is assigned a specific codeword;
training a machine learning core on a plurality of codewords;
processing the plurality of codewords through the trained machine learning core to generate a codeword response;
storing the codeword response and associated processing outputs from the machine learning core in a hierarchical caching system comprising:
a local cache configured to store the codeword responses on an edge device; and
a global cache configured to store the codeword responses across multiple edge devices;
evaluating relevance of the stored codeword responses;
retrieving relevant codeword responses when processing similar prompts; and
translating the codeword response into a response which matches a modality of the prompt.
8 . The method of claim 7 , wherein the machine learning core is a conventional transformer-based architecture comprising:
an embedding layer;
a positional encoding layer; and
a series of transformer layers.
9 . The method of claim 8 , further comprising splitting the codewords into smaller units before processing through the conventional transformer-based architecture.
10 . The method of claim 7 , wherein the machine learning core is a latent transformer core comprising:
a variational autoencoder with an encoder and a decoder; and
a transformer that processes latent space vectors, wherein the transformer does not include an embedding layer and a positional encoding layer.
11 . The method of claim 10 , wherein processing the plurality of codewords through the machine learning core comprises:
generating a plurality of latent space vectors by processing the plurality of codewords through the variational autoencoder's encoder;
learning relationships between the plurality of latent space vectors by processing them through the transformer;
using the learned relationships to generate a plurality of output latent space vectors; and
generating the codeword response by passing the output latent space vectors through the variational autoencoder's decoder.
12 . The method of claim 11 , further comprising splitting the latent space vectors into smaller units before processing through the transformer.