Computerized system and method for distilled deep prediction for personalized stream ranking
The disclosed systems and methods provide a novel framework that provides mechanisms for a Deep & Cross Network (DCN) framework that performs distilled deep prediction for personalized stream ranking on portal websites. The disclosed framework is scalable to satisfy the much more stringent latency and computational requirements required by current network operating environments. The disclosed framework is able to dynamically evaluate and leverage live traffic on network sites in order to provide, update and maintain current recommendations for users as they traverse to a portal and when they navigate within the portal. The disclosed framework implements a DCN model(s) that is capable of being compressed into a model size for a unified optimization within a live traffic environment by combining knowledge distillation and model compression techniques. The disclosed framework is built as a light-weight deep learning model that can be served in production and perform on par with large models.
1. A method comprising:
identifying, by a device, a set of content objects;
analyzing, by the device, the set of content objects, and based on the analysis, determining a set of features;
analyzing, by the device executing a cross network model, the set of features, and determining a cross network model output;
analyzing, by the device executing a deep network model, the set of features, and determining a deep network model output;
concatenating, by the device, the cross network model output and the deep network model output;
training, by the device, a Teacher model based on the concatenation, the Teacher model being a full-scale version of a Deep & Cross Network (DCN) model; and
applying, by the device, a unified optimization loss function that combines probability distribution mapping and logits mapping to compress the Teacher model to a Student model, the Student model being a lightweight version of the Teacher model having a model size optimized for deployment within a live runtime environment to rank a stream of new content objects, wherein the live runtime environment requires ranking responses within a specified latency requirement for incoming user requests.
2. The method of claim 1 , further comprising:
determining a probability distribution based on execution of a neural network softmax function;
determining a mapping between the Teacher model to the Student model based on the probability distribution;
identifying a set of logits for the Teacher model;
mapping, based on a mean square error calculation, the set of logits of the Teacher model to a set of logits for the Student model; and
determining, based on the logits mapping and the probability distribution mapping, a distillation loss, wherein the compression of the Teacher model to the Student model is based on distillation loss.
3. The method of claim 2 , wherein the probability distribution is for a class of content.
4. The method of claim 2 , further comprising:
weighting values of the logits mapping and the probability distribution mapping, wherein the determined distillation loss is based on the weighted values.
5. The method of claim 1 , wherein the cross network model comprises a plurality of layers associated with learned features.
6. The method of claim 1 , wherein the deep network model comprises a plurality of layers comprising functionality for performing neural network overfitting prevention or reduction and batch normalizations.
7. The method of claim 1 , wherein the training is based on a two-way logits of the DCN model.
8. The method of claim 1 , wherein the set of features comprise hand-crafted features.
9. The method of claim 1 , further comprising:
receiving a request for a web page, wherein the live runtime environment corresponds to request;
executing, based on the request, the Student model, and generating a set of content recommendations, the content recommendations being a ranked set of content objects; and
providing the set of content recommendations on the web page.
10. The method of claim 9 , further comprising:
requesting, over the network, third party digital content based at least on information related to the set of content recommendations;
receiving, over the network, the third party digital content; and
communicating, over the network, the third party digital content for display on the web page along with the set of content recommendations.
11. A non-transitory computer-readable storage medium tangibly encoded with computer-executable instructions, that when executed by a processor associated with a device, performs a method comprising:
identifying, by the device, a set of content objects;
analyzing, by the device, the set of content objects, and based on the analysis, determining a set of features;
analyzing, by the device executing a cross network model, the set of features, and determining a cross network model output;
analyzing, by the device executing a deep network model, the set of features, and determining a deep network model output;
concatenating, by the device, the cross network model output and the deep network model output;
training, by the device, a Teacher model based on the concatenation, the Teacher model being a full-scale version of a Deep & Cross Network (DCN) model; and
applying, by the device, a unified optimization loss function that combines probability distribution mapping and logits mapping to compress the Teacher model to a Student model, the Student model being a lightweight version of the Teacher model having a model size optimized for deployment within a live runtime environment to rank a steam of new content objects to rank a stream of new content objects, wherein the live runtime environment requires ranking responses within a specified latency requirement for incoming user requests.
12. The non-transitory computer-readable storage medium of claim 11 , further comprising:
determining a probability distribution based on execution of a neural network softmax function;
determining a mapping between the Teacher model to the Student model based on the probability distribution;
identifying a set of logits for the Teacher model;
mapping, based on a mean square error calculation, the set of logits of the Teacher model to a set of logits for the Student model; and
determining, based on the logits mapping and the probability distribution mapping, a distillation loss, wherein the compression of the Teacher model to the Student model is based on distillation loss.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the probability distribution is for a class of content.
14. The non-transitory computer-readable storage medium of claim 12 , further comprising:
weighting values of the logits mapping and the probability distribution mapping, wherein the determined distillation loss is based on the weighted values.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the cross network model comprises a plurality of layers associated with learned features, wherein the deep network model comprises a plurality of layers comprising functionality for performing neural network overfitting prevention or reduction and batch normalizations.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the training is based on a two-way logits of the DCN model.
17. The non-transitory computer-readable storage medium of claim 11 , further comprising:
receiving a request for a web page, wherein the live runtime environment corresponds to request;
executing, based on the request, the Student model, and generating a set of content recommendations, the content recommendations being a ranked set of content objects; and
providing the set of content recommendations on the web page.
18. A computing device comprising:
a processor configured to:
identify a set of content objects;
analyze the set of content objects, and based on the analysis, determine a set of features;
analyze, by executing a cross network model, the set of features, and determine a cross network model output;
analyze, executing a deep network model, the set of features, and determine a deep network model output;
concatenate the cross network model output and the deep network model output;
train a Teacher model based on the concatenation, the Teacher model being a full-scale version of a Deep & Cross Network (DCN) model; and
apply a unified optimization loss function that combines probability distribution mapping and logits mapping to compress the Teacher model to a Student model, the Student model being a lightweight version of the Teacher model having a model size optimized for deployment within a live runtime environment to rank a steam of new content objects to rank a stream of new content objects, wherein the live runtime environment requires ranking responses within a specified latency requirement for incoming user requests.
19. The computing device of claim 18 , further comprising:
determine a probability distribution based on execution of a neural network softmax function;
determine a mapping between the Teacher model to the Student model based on the probability distribution;
identify a set of logits for the Teacher model;
map, based on a mean square error calculation, the set of logits of the Teacher model to a set of logits for the Student model; and
determine, based on the logits mapping and the probability distribution mapping, a distillation loss, wherein the compression of the Teacher model to the Student model is based on distillation loss.
20. The computing device of claim 18 , further comprising:
receive a request for a web page, wherein the live runtime environment corresponds to request;
execute, based on the request, the Student model, and generate a set of content recommendations, the content recommendations being a ranked set of content objects; and
provide the set of content recommendations on the web page.