Transformers adapted for predictive targeting based on web browsing history
Some embodiments described herein relate to computer implemented methods and/or memory (e.g., a non-transitory, processor-readable medium) storing code configured to be executed by a processor to cause the processor to perform the method. The method can include training a first transformer with a plurality of website visitation records to predict a missing website in a sequence of websites. The first transformer can include a foundation model and a first prediction head. A second transformer model can be trained with a plurality of conversion event data to predict a probability of a conversion event occurring. The second transformer model can include the foundation model and a second prediction head.
1. A non-transitory, processor-readable medium storing code configured to be executed by a processor, the code comprising instructions to cause the processor to:
train a first transformer model with a plurality of website visitation records to predict a missing website in a sequence of websites based on a sequence of locations in a p-dimensional embedding, each location from the sequence of locations associated with a website represented in the plurality of website visitation records, the first transformer including a foundation model and a first prediction head; and
train a second transformer model with a plurality of conversion event data to predict a probability of a conversion event occurring, the second transformer model including the foundation model and a second prediction head.
2. The non-transitory, processor-readable medium of claim 1 , wherein an architecture of the first transformer model was developed for natural language processing tasks.
3. The non-transitory, processor-readable medium of claim 1 , wherein the plurality of website visitation records includes at least one of store visitation history data, device usage data, or application usage history.
4. The non-transitory, processor-readable medium of claim 1 , wherein each website represented in the plurality of website visitation records is represented by a vector associated with a location in a p-dimensional embedding space.
5. The non-transitory, processor-readable medium of claim 1 , wherein each website visitation record of the plurality of website visitation records includes a timestamp indicating a time a website associated with that website visitation record was visited.
6. The non-transitory, processor-readable medium of claim 1 , the code further comprising code to cause the processor to encode each website from the sequence of websites into a vector associated with a location in the p-dimensional embedding.
7. The non-transitory, processor-readable medium of claim 6 , wherein:
the first transformer model is configured to receive the sequence of websites;
the sequence of websites has an empty position; and
the first transformer model is configured to output a prediction of a website in the empty position based on the location of each website from the sequence of websites in the p-dimensional embedding.
8. The non-transitory, processor-readable medium of claim 1 , the code further comprising code to cause the processor to:
receive an indication of a sequence of website visitations; and
predict the probability of the conversion event occurring based on the sequence of website visitations using the second transformer model.
9. The non-transitory, processor-readable medium of claim 1 , the code further comprising code to cause the processor to:
receive an indication of a sequence of website visitations;
encode each website included in the sequence of websites into a vector from a plurality of vectors, each vector from the plurality of vectors associated with a location in the p-dimensional embedding; and
predict a probability of the conversion event occurring based on the sequence of the plurality of vectors.
10. A computer-implemented method, comprising:
accessing a first dataset representing a first set of website visitation records;
accessing a second dataset representing a second set of website visitation records;
training a first transformer model using the first dataset to predict a website in an empty position of a sequence of websites, each website from the sequence of websites defining a point in a p-dimensional embedding, the first transformer model including a foundation model and a first prediction head;
training a second transformer model using the second dataset, the second transformer model including the foundation model and a second prediction head;
receiving a third dataset representing a sequence of websites; and
predicting a probability of a conversion event based on the third data set and using the second transformer model.
11. The method of claim 10 , wherein each website represented in the first data set, the second data set, and the third data set is represented by a vector associated with a location in the p-dimensional embedding space.
12. The method of claim 10 , wherein each data point of the first dataset includes a timestamp indicating a time associated with generating that data point.
13. The method of claim 10 , wherein the second dataset includes data records indicating conversion event data.
14. The method of claim 10 , further comprising:
receiving an ordered sequence of website visitation records; and
applying the second transformer model to the ordered sequence of website visitation records to predict a probability of the conversion event occurring based on the ordered sequence of website visitation records.
15. The method, of claim 10 , further comprising:
masking a website in the first dataset, to train the first transformer model, training the first transformer model includes comparing the website the first transformer model predicts in the empty position to a target website, the target website being the masked website.
16. The method of claim 15 , wherein training the first transformer model includes:
evaluating a difference between the website the first transformer model predicts in the empty position and the target website using a loss function; and
adjusting a parameter of the first transformer model when the loss function does not meet a predefined threshold.
17. The method of claim 15 , wherein training the first transformer model includes:
evaluating a difference between a website the first transformer model predicts in the empty position and the target website using a loss function; and
adjusting a first parameter of the first transformer model when the loss function does not meet a predefined threshold while holding a second parameter of the first transformer model constant.
18. The method of claim 15 , wherein the first parameter is associated with at least one of an encoder block or another block of the first transformer model and the first parameter is adjusted using backpropagation.