Edge cloud hierarchical language model design
The present disclosure relates to systems and methods for using language models in locations with limited network connectivity. The systems and methods include a hierarchical edge architecture with a plurality of language models with diverse compute capabilities. The systems and methods dynamically select a language model from the plurality of language models to use to respond to a query received by a user in response to determining a level of network connectivity available at a user device.
1 . A method comprising:
receiving, at a user device, a query;
selecting, from a plurality of language models with diverse compute capabilities, a small language model (SLM) on the user device based on data available on the user device, query parameters of the query, a required accuracy for the query, and available network connectivity;
retrieving user context and industry context relevant to the query from the data on the user device, wherein the user context and the industry context is automatically generated from collected data and provided in vectorized databases for storage on the user device;
using the SLM to provide a response to the query tailored to the user context and the industry context; and
providing the response to a user.
2 . The method of claim 1 , wherein the plurality of language models include the SLM at the user device, a medium language model at a device of a back office edge in communication with the user device, and a large language model (LLM) at a device of a cloud network in communication with the user device.
3 . The method of claim 2 , wherein the SLM has less compute and memory footprint as compared to the medium language model and the LLM, and the medium language model has less compute and memory footprint as compared to the LLM.
4 . The method of claim 1 , further comprising:
selecting, from the plurality of language models, a medium language model on a device at a back office edge in communication with the user device to provide the response to the query in response to determining that the SLM is unable to provide the response to the query within a latency threshold, wherein the medium language model includes more compute and memory footprint as compared to the SLM.
5 . The method of claim 1 , further comprising:
selecting, from the plurality of language models, a large language model (LLM) at a cloud network in communication with the user device to provide the response to the query in response to determining that the SLM is unable to provide an accurate response to the query, wherein the LLM includes more compute and memory footprint as compared to the SLM.
6 . The method of claim 1 , further comprising:
using, by the SLM, the user context and the industry context stored on the user device to provide the response to the query.
7 . The method of claim 6 , wherein the user context includes information obtained from private documents of the user with data specific to the user and the response is tailored to the user using the data specific to the user.
8 . The method of claim 6 , wherein the industry context includes data specific to an industry that is related to the user.
9 . The method of claim 6 , further comprising:
periodically receiving the context from a device at a back office edge in communication with the user device or devices on a cloud network in communication with the user device, wherein the context is obtained from external sources and information specific to the user.
10 . The method of claim 6 , wherein the user context and the industry context is automatically generated by a device at a back office edge in communication with the user device or devices on a cloud network in communication with the user device in response to sensory data obtained by sensors at a location.
11 . A user device comprising:
a memory to store data and instructions; and
a processor operable to communicate with the memory, wherein the processor is operable to:
receive a query;
select, from a plurality of language models with diverse compute capabilities, a small language model (SLM) on the user device based on data available on the user device, query parameters of the query, a required accuracy for the query, and available network connectivity;
retrieving user context and industry context relevant to the query from the data on the user device, wherein the user context and the industry context is automatically generated from collected data and provided in vectorized databases for storage on the user device;
use the SLM on the user device to provide a response to the query tailored to the user context and the industry context; and
provide the response to a user.
12 . The user device of claim 11 , wherein the plurality of language models include the SLM at the user device, a medium language model at a device of a back office edge in communication with the user device, and a large language model (LLM) at a device of a cloud network in communication with the user device.
13 . The user device of claim 12 , wherein the SLM has less compute as compared to the medium language model and the LLM, and the medium language model has more compute as compared to the SLM and less compute as compared to the LLM.
14 . The user device of claim 11 , wherein the processor is further operable to select, from the plurality of language models, a medium language model on a device at a back office edge in communication with the user device to provide the response to the query in response to determining that the SLM is unable to provide the response to the query within a latency threshold, wherein the medium language model includes more compute as compared to the SLM.
15 . The user device of claim 11 , wherein the processor is further operable to select, from the plurality of language models, a large language model (LLM) at a cloud network in communication with the user device to provide the response to the query in response to determining that the SLM is unable to provide an accurate response to the query, wherein the LLM includes more compute as compared to the SLM.
16 . The user device of claim 11 , wherein the processor is further operable to use, by the SLM, the user context and the industry context stored on the user device to provide the response to the query.
17 . The user device of claim 16 , wherein the context includes user context with data specific to the user obtained from images of a location of the user and the response is tailored to the user using the data specific to the user.
18 . The user device of claim 16 , wherein the industry context includes data specific to an industry that is related to the user.
19 . The user device of claim 16 , wherein the processor is further operable to periodically receive the context from a device at a back office edge in communication with the user device or devices on a cloud network in communication with the user device, wherein the context is obtained from external sources and information specific to the user.
20 . The user device of claim 16 , wherein the context is automatically generated by a device at a back office edge in communication with the user device or devices on a cloud network in communication with the user device in response to sensory data obtained by sensors at a location; and the context is provided to the user device for storage in vectorized databases.