Ideographic contrastive autoencoder for large language model fine-tuning
Ideographic contrastive autoencoder for large language model fine-tuning is disclosed, including: obtaining a set of user activities according to a specified task; obtaining respective sets of input features from the set of user activities; using an encoder network of an autoencoder to encode the respective sets of input features into a set of words; prompting a machine learning model to perform the specified task using the set of words, wherein the machine learning model has been fine-tuned using a custom lexicographical vocabulary associated with the autoencoder; and presenting, at a user interface, a message determined based at least in part on an output result from the machine learning model.
1 . A system, comprising:
one or more processors configured to:
obtain a set of user activities according to a specified task;
obtain respective sets of input features from the set of user activities, wherein a set of input features comprises a set of structured data associated with an input activity;
use an encoder network of an autoencoder to encode the respective sets of input features into a set of input words, wherein the set of input features is encoded by the encoder network into a fixed-length series of numbers, wherein the fixed-length series of numbers is mapped into a set of non-numerical characters included in a selected character space associated with a custom lexicographical vocabulary, wherein the set of non-numerical characters forms an input word that comprises an abstraction of the set of structured data associated with the input activity;
send a prompt to a machine learning model to cause the machine learning model to perform the specified task using the set of input words, wherein the machine learning model has been fine-tuned using pairs of words encoded by the autoencoder from reference sets of input features associated with reference activities and reference text-based descriptions of the reference sets of input features associated with the reference activities, wherein the machine learning model has been fine-tuned to understand that the input word represents the set of structured data associated with the input activity, wherein the input word represents the set of structured data associated with the input activity in the prompt in fewer tokens than would have been needed by the set of structured data itself, wherein the input word is more efficient to compute by the machine learning model during inference than the set of structured data represented by the input word;
receive an output result from the machine learning model;
parse the output result to determine whether the output result includes an output word;
in response to a determination that the output result includes the output word, input the output word into a decoder network of the autoencoder to obtain a reconstructed set of features associated with an output activity; and
present, at a user interface, a message associated with the output activity determined based at least in part on the reconstructed set of features associated with the output activity and the output result; and
one or more memories coupled to the one or more processors and configured to provide instructions to the one or more processors.
2 . The system of claim 1 , wherein the machine learning model comprises a large language model (LLM).
3 . The system of claim 1 , wherein character positions within the input word are associated with different weights.
4 . The system of claim 1 , wherein the one or more processors are further configured to:
programmatically convert the reconstructed set of features associated with the output activity into a text-based description of one or more activities, wherein the message comprises the text-based description and the output result.
5 . The system of claim 1 , wherein the specified task comprises prediction of a subsequent user activity using the set of user activities, wherein the output activity comprises the subsequent user activity.
6 . The system of claim 1 , wherein the specified task further comprises summarization of the set of user activities.
7 . The system of claim 1 , wherein the specified task further comprises generation of a personalized workout plan.
8 . The system of claim 1 , wherein the specified task further comprises generation of a personalized recommended route associated with the output activity.
9 . The system of claim 1 , wherein the specified task comprises determination of an anomalous user activity, wherein the output activity comprises the anomalous user activity.
10 . The system of claim 1 , wherein the specified task further comprises determination of a cause of injury.
11 . The system of claim 1 , wherein the fixed-length series of numbers comprises a fixed-length series of real numbers, and wherein each real number of the fixed-length series of real numbers is rounded to a nearest integer prior to being mapped to a corresponding non-numerical character included in the custom lexicographical vocabulary.
12 . The system of claim 1 , wherein the selected character space associated with the custom lexicographical vocabulary is not associated with the English language.
13 . A system, comprising:
one or more processors configured to:
obtain a set of training data from a set of user activity data, wherein the set of training data comprises a first set of input features comprising a first set of structured data associated with a first activity;
train an autoencoder using the set of training data to obtain a custom lexicographical vocabulary, wherein the autoencoder is trained to encode the first set of input features into a fixed-length series of numbers, wherein the fixed-length series of numbers is mapped into a set of non-numerical characters included in a selected character space associated with the custom lexicographical vocabulary, wherein the set of non-numerical characters forms a first word that comprises an abstraction of the first set of structured data associated with the first activity; and
fine-tune a machine learning model to;
output a text-based description of a second set of input features associated with a second word encoded by the autoencoder from the second set of input features associated with a second activity; and
perform a task in response to a prompt including a third word, wherein the machine learning model is fine-tuned to understand that the third word represents a third set of structured data associated with a third activity, wherein the third word represents the third set of structured data associated with the third activity in the prompt to the machine learning model in fewer tokens than would have been needed by the third set of structured data itself,
wherein the third word is more efficient to compute by the machine learning model during inference than the third set of structured data represented by the third word,
wherein an output result from the machine learning model is parsed to determine whether the output result includes an output word that is to be input into a decoder network of the autoencoder to obtain a reconstructed set of features associated with an output activity; and
one or more memories coupled to the one or more processors and configured to provide instructions to the one or more processors.
14 . The system of claim 13 , wherein the first set of input features includes one or more of the following: an average speed, a maximum speed, a distance, an elevation gain, a moving time, an elapsed time, an average grade, a maximum grade, an average cadence, a maximum cadence, an average heart rate, a maximum heart rate, whether the first activity was part of a race or not, and whether the first activity was part of a commute or not.
15 . The system of claim 13 , wherein the autoencoder is configured to output encoded words of the selected character space from a specified numerical range.
16 . The system of claim 13 , wherein to train the autoencoder comprises to:
receive a pair of a reference set of input features from the set of training data and a modified set of input features derived from the reference set of input features;
input the reference set of input features into an encoder network of the autoencoder to obtain a reference word from a bottleneck layer of the autoencoder;
input the modified set of input features into the encoder network of the autoencoder to obtain a fuzzy word from the bottleneck layer of the autoencoder; and
determine a similarity penalty based at least in part on a comparison between the reference word and the fuzzy word.
17 . The system of claim 16 , wherein the comparison comprises a determined Euclidean distance between the reference word and the fuzzy word.
18 . The system of claim 16 , wherein the reconstructed set of features associated with the output activity comprises a first reconstructed set of features, and wherein to train the autoencoder further comprises to:
input the reference word into the decoder network of the autoencoder to obtain a second reconstructed set of features; and
determine a reconstruction penalty based at least in part on a comparison between the reference set of input features and the second reconstructed set of features.
19 . The system of claim 18 , wherein to train the autoencoder further comprises to:
determine an aggregated penalty based at least in part on the similarity penalty and the reconstruction penalty; and
update parameters of the autoencoder based at least in part on the aggregated penalty.
20 . The system of claim 13 , wherein to fine-tune the machine learning model further comprises to:
receive an encoded word encoded from a reference set of input features associated with a reference user activity from the set of training data and a reference text-based description of the reference set of input features;
prompt the machine learning model using the encoded word to obtain an output description;
determine a domain adaptation penalty based on a comparison between a reference text-based description of the reference user activity and the output description; and
update parameters of the machine learning model based at least in part on the domain adaptation penalty.
21 . The system of claim 20 , wherein the output result comprises a first output result, and wherein to fine-tune the machine learning model further comprises to:
receive a set of words from reference input features associated with user activities from the set of training data and a reference text-based result of performing a specified task on the set of words;
prompt the machine learning model to perform the specified task using the set of words to obtain a second output result;
determine a task-specific penalty based at least in part on a comparison between the reference text-based result and the second output result; and
update the parameters of the machine learning model based at least in part on the task-specific penalty.
22 . The system of claim 13 , wherein the selected character space associated with the custom lexicographical vocabulary is not associated with the English language.