Methods, mediums, and systems for representing a model in a memory of device
Exemplary embodiments relate to methods, mediums, and systems for moving language models from a server to the client device. Such embodiments may be deployed in an environment where the server is not able to provide modeling services to the clients, such as an end-to-end encrypted (E2EE) environment. Several different techniques are described to address issues of size and complexity reduction, model architecture optimization, model training, battery power reduction, and latency reduction.
1. A method comprising:
accessing a model configured to be executed as a plurality of submodels, the submodels comprising a first submodel and a second submodel;
loading the first submodel into a memory of a device;
generating a first output from the first submodel;
determining to proceed with processing the model based on the first output;
in response to the determining, removing the first submodel from the memory and loading the second submodel into the memory after the first submodel is removed to continue executing the model; and
generating a second output from the second submodel.
2. The method of claim 1 , wherein the first output is executed before the second submodel based on the first submodel screening out more options than the second submodel or based on the first submodel being more likely to terminate processing than the second submodel.
3. The method of claim 2 , wherein the first submodel is a personalization model that determines whether or not a given user is likely to use a recommendation output by the model.
4. The method of claim 1 , wherein the first submodel is independent of calculations performed by the second submodel.
5. The method of claim 1 , wherein the model is a natural language understanding model.
6. The method of claim 1 , wherein the device is an end-user device in an end-to-end encrypted environment.
7. The method of claim 1 , wherein a device executing the model exhibits reduced power usage as compared to the same device loading an entirety of the model into memory at once.
8. A non-transitory computer-readable medium storing instructions configured to be executed by a processor to cause the processor to:
access a model configured to be executed as a plurality of submodels, the submodels comprising a first submodel and a second submodel;
load the first submodel into a memory of a device;
generate a first output from the first submodel;
determine to proceed with processing the model based on the first output;
in response to the determining, remove the first submodel from the memory and loading the second submodel into the memory after the first submodel is removed to continue executing the model; and
generate a second output from the second submodel.
9. The medium of claim 8 , wherein the first output is executed before the second submodel based on the first submodel screening out more options than the second submodel or based on the first submodel being more likely to terminate processing than the second submodel.
10. The medium of claim 9 , wherein the first submodel is a personalization model that determines whether or not a given user is likely to use a recommendation output by the model.
11. The medium of claim 8 , wherein the first submodel is independent of calculations performed by the second submodel.
12. The medium of claim 8 , wherein the model is a natural language understanding model.
13. The medium of claim 8 , wherein the device is an end-user device in an end-to-end encrypted environment.
14. The medium of claim 8 , wherein a device executing the model exhibits reduced power usage as compared to the same device loading an entirety of the model into memory at once.
15. An apparatus comprising:
a non-transitory computer-readable medium storing a model configured to be executed as a plurality of submodels, the submodels comprising a first submodel and a second submodel;
a memory configured to support the plurality of submodels during execution; and
a hardware processor configured to:
load the first submodel into the memory;
generate a first output from the first submodel;
determine to proceed with processing the model based on the first output;
in response to the determining, remove the first submodel from the memory and loading the second submodel into the memory after the first submodel is removed to continue executing the model; and
generate a second output from the second submodel.
16. The apparatus of claim 15 , wherein the first output is executed before the second submodel based on the first submodel screening out more options than the second submodel or based on the first submodel being more likely to terminate processing than the second submodel.
17. The apparatus of claim 16 , wherein the first submodel is a personalization model that determines whether or not a given user is likely to use a recommendation output by the model.
18. The apparatus of claim 15 , wherein the first submodel is independent of calculations performed by the second submodel.
19. The apparatus of claim 15 , wherein the model is a natural language understanding model.
20. The apparatus of claim 15 , wherein the apparatus is an end-user device in an end-to-end encrypted environment.