Nested machine learning architecture
In one embodiment, a method includes a preprocessing stage of a neural network model, where the preprocessing stage includes first and second preprocessing modules. Each of the two modules has first input that may receive a dense input and a second input that may receive a sparse input. Each module generates latent vector representations of their respective first and second inputs, and combine the latent vectors with the original first input to define an intermediate output. The intermediate output of the first module is fed into the first input of the second module.
1. A method for generating a personalized prediction or ranking for a task using a neural network model, comprising:
by a computing device, receiving a first data and a second data associated with a user;
by the computing device, using the neural network model to process the first data and the second data, the neural network model having a plurality of modules, including a first module and a second module, wherein the processing comprises:
providing the first data and the second data to the first module for processing by the first module, wherein the first data includes user information associated with the user, the second data includes semantic information related to the task, and the processing by the first module comprises:
receiving a first input and a second input, wherein the first input comprises the first data and the second input comprises the second data;
generating a first latent vector representation of the first input and a second latent vector representation of the second input;
modeling a pairwise interaction between the first latent vector representation and the second latent vector representation; and
generating an output based on a combination of the pairwise interaction and the first input; and
after generating the output of the first module by processing the first data and the second data using the first module, providing the output of the first module and the second data to the second module for processing by the second module, wherein the processing by the second module comprises:
receiving a first input of the second module and a second input of the second module, wherein the first input of the second module comprises the output of the first module and the second input of the second module comprises the second data;
generating a second-module first latent vector representation of the first input of the second module and a second-module second latent vector representation of the second input of the second module;
modeling a second-module pairwise interaction between the second-module first latent vector representation and the second-module second latent vector representation; and
generating a second-module output based on a combination of the second-module pairwise interaction and the first input of the second module.
2. The method of claim 1 , wherein the output of the first module is directly fed forward to the second module as the first input of the second module.
3. The method of claim 1 , wherein the neural network model further has a plurality of layers of nodes, at least one layer of nodes is between the first module and the second module in a feedforward arrangement, and the output of the first module is fed forward to the second module as the first input of the second module via the at least one layer of nodes.
4. The method of claim 1 , wherein each of the plurality of modules is configured to:
receive a respective first input and a respective plurality of second inputs;
generate a respective first latent vector representation of the respective first input and a respective plurality of second latent vector representations respectively corresponding to its respective plurality of second inputs; and
model pairwise interactions between unique pairwise combinations of the respective first latent vector representation of the module and the respective plurality of second latent vector representations of the module.
5. The method of claim 4 , wherein:
the user information associated with the user includes observably determinable data characterizing a network session associated with the user;
the respective plurality of second inputs of each of the plurality of modules provide contextual information related to network options available to the user; and
the neural network model provides a prediction related to the network user.
6. The method of claim 5 , wherein the prediction is a probability of the user interacting with specific content within a webpage.
7. The method of claim 4 , wherein in each of said plurality of modules:
the respective plurality of said second latent vector representations are generated based on embeddings associated with its respective plurality of said second inputs, the embeddings and its respective first latent vector being of equal dimensions.
8. The method of claim 7 , wherein the pairwise interactions modeled by each of the plurality of modules are modeled by dot product operations or cosine similarity operations.
9. The method of claim 7 , wherein the respective first vector representation is obtained by linear transformation of the respective first input.
10. The method of claim 7 , wherein the respective first vector representation is obtained by use of an embedding associated with the respective first input.
11. The method of claim 7 , wherein for each of the plurality of modules:
its respective first input is a dense feature vector;
its respective second inputs are sparse feature vectors, each consisting of a list of data items; and
the generating of the plurality of said second latent vector representations includes, for each sparse feature vector, assigning a low dimensional vector representation to each data item in the list of data items of the sparse feature vector by use of an embedding matrix to define a group of low dimensional vector representations, assigning a higher weight to low dimensional vector representations of data items determined to be of higher relevance to a current context than to low dimensional vector representations of data items determined to be of lesser relevance to the current context, combining the group of low dimensional vector representations, including their respective weights, by an element-wise pooling process to define the embedding of the sparse feature vector.
12. The method of claim 11 , wherein one of the sparse feature vectors corresponds to a list of webpage identifiers identifying webpages with which a network user has interacted, and webpage identifiers identifying webpages having a higher frequency of interaction with the network user are determined to be of higher relevance to the current context and are assigned weights higher than webpage identifiers identifying webpages having a lower frequency of interaction with the network user.
13. The method of claim 11 , wherein the data items have coefficient scores based on a network user's engagement history with the data items, and the relevance of a data item to the current context is determined from its coefficient score.
14. The method of claim 11 , wherein the data items in at least one of the list of data items are defined by use of a plurality of categorizing trees, one categorizing tree per data item, wherein an output of each categorizing tree is dependent upon a data element in the dense feature vector that is the first input of the first module.
15. The method of claim 1 , wherein the first module and the second module receive the same second input.
16. The method of claim 15 , wherein the first module and the second module assign a different second latent vector representation to their respective, same second input.
17. One or more computer-readable non-transitory storage media embodying software that is operable for generating a personalized prediction or ranking for a task using a neural network model when executed to:
receive a first data and a second data associated with a user;
use the neural network model to process the first data and the second data, the neural network model having a plurality of modules, including a first module and a second module, wherein the processing comprises:
providing the first data and the second data to the first module for processing by the first module, wherein the first data includes user information associated with the user, the second data includes semantic information related to the task, and the processing by the first module comprises:
receiving a first input and a second input, wherein the first input comprises the first data and the second input comprises the second data;
generating a first latent vector representation of the first input and a second latent vector representation of the second input;
modeling a pairwise interaction between the first latent vector representation and the second latent vector representation; and
generating an output based on a combination of the pairwise interaction and the first input; and
after generating the output of the first module by processing the first data and the second data using the first module, providing the output of the first module and the second data to the second module for processing by the second module, wherein the processing by the second module comprises:
receiving a first input of the second module and a second input of the second module, wherein the first input of the second module comprises the output of the first module and the second input of the second module comprises the second data;
generating a second-module first latent vector representation of the first input of the second module and a second-module second latent vector representation of the second input of the second module;
modeling a second-module pairwise interaction between the second-module first latent vector representation and the second-module second latent vector representation; and
generating a second-module output based on a combination of the second-module pairwise interaction and the first input of the second module.
18. The media of claim 17 , wherein the neural network model further has a plurality of layers of nodes, at least one layer of nodes is between the first module and the second module in a feedforward arrangement, and the output of the first module is fed forward to the second module as the first input of the second module via the at least one layer of nodes.
19. A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable for generating a personalized prediction or ranking for a task using a neural network model when executed by one or more of the processors to cause the system to:
receive a first data and a second data associated with a user;
use the neural network model to process the first data and the second data, the neural network model having a plurality of modules, including a first module and a second module, wherein the processing comprises:
providing the first data and the second data to the first module for processing by the first module, wherein the first data includes user information associated with the user, the second data includes semantic information related to the task, and the processing by the first module comprises:
receiving a first input and a second input, wherein the first input comprises the first data and the second input comprises the second data;
generating a first latent vector representation of the first input and a second latent vector representation of the second input;
modeling a pairwise interaction between the first latent vector representation and the second latent vector representation; and
generating an output based on a combination of the pairwise interaction and the first input; and
after generating the output of the first module by processing the first data and the second data using the first module, providing the output of the first module and the second data to the second module for processing by the second module, wherein the processing by the second module comprises:
receiving a first input of the second module and a second input of the second module, wherein the first input of the second module comprises the output of the first module and the second input of the second module comprises the second data;
generating a second-module first latent vector representation of the first input of the second module and a second-module second latent vector representation of the second input of the second module;
modeling a second-module pairwise interaction between the second-module first latent vector representation and the second-module second latent vector representation; and
generating a second-module output based on a combination of the second-module pairwise interaction and the first input of the second module.
20. The system of claim 19 , wherein the neural network model further has a plurality of layers of nodes, at least one layer of nodes is between the first module and the second module in a feedforward arrangement, and the output of the first module is fed forward to the second module as the first input of the second module via the at least one layer of nodes.