Graph structure aware incremental learning for recommender system

1 . A method for training a recommender system (RS) that is configured to make recommendations in respect of a bipartite graph that comprises a plurality of user nodes, a plurality of item nodes, and an observed graph topology that defines edges connecting at least some of the user nodes to some of the item nodes, the RS including an existing graph neural network (GNN) model configured by an existing set of parameters, the method comprising:

applying a loss function to compute an updated set of parameters for an updated GNN model that is trained with a new graph using the existing set of parameters as initialization parameters, the loss function being configured to distil knowledge based on node embeddings generated by the existing GNN model in respect of an existing graph, wherein the new graph includes a plurality of user nodes and a plurality of item nodes that are also included in the existing graph, wherein the loss function includes a local structure distillation component that is configured to distil, during an iterative training process, a local graph structure for the existing graph for at least some item nodes and user nodes that are included in both the existing graph and the new graph; and

determining the local structure distillation component, wherein the determining comprises:

for each of the at least some of the user nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of item nodes in the existing graph for the user node;

determining an existing average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the updated GNN model;

determining a first user value that is a dot product of: (i) an embedding generated for the user node by the existing GNN model and (ii) the existing average local neighborhood user node embedding for the user node;

determining a second user value that is a dot product of: (i) an embedding generated for the user node by the updated GNN model and (ii) the new average local neighborhood user node embedding for the user node; and

determining a user node difference between the first user value and the second user value;

determining a user node average distance value that is an average of the user node difference determined in respect of the at least some of the user nodes;

for each of the at least some of the item nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of user nodes in the existing graph for the item node;

determining an existing average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the updated GNN model;

determining a first item value that is a dot product of: (i) an embedding generated for the item node by the existing GNN model and (ii) the existing average local neighborhood item node embedding for the item node;

determining a second item value that is a dot product of: (i) an embedding generated for the item node by the updated GNN model and (ii) the new average local neighborhood item node embedding for the item node; and

determining an item node difference between the first item value and the second item value;

determining an item node average distance value that is an average of the item node difference determined in respect of the at least some of the item nodes;

wherein the local structure distillation component is based on a sum of the user node average distance value and the item node average distance value; and

replacing the existing GNN model of the RS with the updated GNN model.

2 . The method of claim 1 wherein during the iterative training process, interim sets of updated parameters are generated for training the updated GNN model, and the updated GNN model is configured by every interim set of updated parameters to generate interim node embeddings in respect of the new graph.

3 . The method of claim 1 wherein the local structure distillation component comprises a product of a local distillation hyper-parameter that is configured to control a magnitude of the local graph structure distillation component and the sum of the user node average distance value and the item node average distance value.

4 . The method of claim 1 wherein the loss function includes a global structure distillation component that is configured to distil, during the iterative training process, a global graph structure for the existing graph for the at least some item nodes and user nodes that are included in both the existing graph and the new graph.

5 . The method of claim 4 wherein the method comprises determining the global structure distillation component by:

determining, for each of the at least some user nodes and item nodes, a structure similarity between the existing graph and the new graph based on node embeddings generated by the existing GNN model and the updated GNN model; and

determining, based on the determined structure similarities, global structure distributions for the existing graph and the new graph;

wherein the global structure distillation component is based on Kullback-Leibler (KL) divergences between the global structure distributions for the existing graph and the new graph.

6 . The method of claim 5 wherein the global structure distillation component is based on a global distillation hyper-parameter configured to control a magnitude of the global graph structure distillation component.

7 . The method of claim 1 wherein the loss function includes a self-embedding distillation component that is configured to preserve, during the iterative training process, knowledge from the existing graph for the at least some item nodes and user nodes that are included in both the existing graph and the new graph.

8 . The method of claim 1 wherein the loss function includes Bayesian personalized ranking (BPR) loss component.

9 . A processing system for implementing a recommender system (RS) that is configured to make recommendations in respect of a bipartite graph that comprises a plurality of user nodes, a plurality of item nodes, and an observed graph topology that defines edges connecting at least some of the user nodes to some of the item nodes, the RS including an existing graph neural network (GNN) model configured by an existing set of parameters, the processing system comprising:

a processing device and a non-volatile storage coupled to the processing device and storing executable instructions that when executed by the processing device cause the processing system to perform a method including:

applying a loss function to compute an updated set of parameters for an updated GNN model that is trained with a new graph using the existing set of parameters as initialization parameters, the loss function being configured to distil knowledge based on node embeddings generated by the existing GNN model in respect of an existing graph, wherein the new graph includes a plurality of user nodes and a plurality of item nodes that are also included in the existing graph, wherein the loss function includes a local structure distillation component that is configured to distil, during an iterative training process, a local graph structure for the existing graph for at least some item nodes and user nodes that are included in both the existing graph and the new graph; and

determining the local structure distillation component, wherein the determining comprises:

for each of the at least some of the user nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of item nodes in the existing graph for the user node;

determining an existing average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the updated GNN model;

determining a first user value that is a dot product of: (i) an embedding generated for the user node by the existing GNN model and (ii) the existing average local neighborhood user node embedding for the user node;

determining a second user value that is a dot product of: (i) an embedding generated for the user node by the updated GNN model and (ii) the new average local neighborhood user node embedding for the user node; and

determining a user node difference between the first user value and the second user value;

determining a user node average distance value that is an average of the user node difference determined in respect of the at least some of the user nodes;

for each of the at least some of the item nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of user nodes in the existing graph for the item node;

determining an existing average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the updated GNN model;

determining a first item value that is a dot product of: (i) an embedding generated for the item node by the existing GNN model and (ii) the existing average local neighborhood item node embedding for the item node;

determining a second item value that is a dot product of: (i) an embedding generated for the item node by the updated GNN model and (ii) the new average local neighborhood item node embedding for the item node; and

determining an item node difference between the first item value and the second item value;

determining an item node average distance value that is an average of the item node difference determined in respect of the at least some of the item nodes;

wherein the local structure distillation component is based on a sum of the user node average distance value and the item node average distance value; and

replacing the existing GNN model of the RS with the updated GNN model.

10 . The system of claim 9 wherein during the iterative training process, interim sets of updated parameters are generated for training the updated GNN model, and the updated GNN model is configured by every interim set of updated parameters to generate interim node embeddings in respect of the new graph.

11 . The system of claim 9 wherein the local structure distillation component comprises a product of a local distillation hyper-parameter that is configured to control a magnitude of the local graph structure distillation component and the sum of the user node average distance and the item node average distance.

12 . The system of claim 10 wherein the loss function includes a global structure distillation component that is configured to distil, during the iterative training process, a global graph structure for the existing graph for the at least some item nodes and user nodes that are included in both the existing graph and the new graph.

13 . The system of claim 12 wherein the executable instructions cause the processing system to determine the global structure distillation component by:

determining, for each of the at least some user nodes and item nodes, a structure similarity between the existing graph and the new graph based on node embeddings generated by the existing GNN model and the updated GNN model; and

determining, based on the determined structure similarities, global structure distributions for the existing graph and the new graph;

wherein the global structure distillation component is based on Kullback-Leibler (KL) divergences between the global structure distributions for the existing graph and the new graph.

14 . The system of claim 13 wherein the global structure distillation component is based on a global distillation hyper-parameter configured to control a magnitude of the global graph structure distillation component.

15 . The system of claim 9 wherein the loss function includes a self-embedding distillation component that is configured to preserve, during the iterative training process, knowledge from the existing graph for the at least some item nodes and user nodes that are included in both the existing graph and the new graph, and the loss function includes Bayesian personalized ranking (BPR) loss component.

16 . A non-volatile computer readable memory storing executable instructions for implementing a recommender system (RS) that is configured to make recommendations in respect of a bipartite graph that comprises a plurality of user nodes, a plurality of item nodes, and an observed graph topology that defines edges connecting at least some of the user nodes to some of the item nodes, the RS including an existing graph neural network (GNN) model configured by an existing set of parameters, the executable instructions including instructions to configure a processing system to perform a method including:

applying a loss function to compute an updated set of parameters for an updated GNN model that is trained with a new graph using the existing set of parameters as initialization parameters, the loss function being configured to distil knowledge based on node embeddings generated by the existing GNN model in respect of an existing graph, wherein the new graph includes a plurality of user nodes and a plurality of item nodes that are also included in the existing graph, wherein the loss function includes a local structure distillation component that is configured to distil, during an iterative training process, a local graph structure for the existing graph for at least some item nodes and user nodes that are included in both the existing graph and the new graph; and

determining the local structure distillation component, wherein the determining comprises:

for each of the at least some of the user nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of item nodes in the existing graph for the user node;

determining an existing average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood user node embedding for the user node based on an average of embeddings generated for the item nodes in the neighborhood set by the updated GNN model;

determining a first user value that is a dot product of: (i) an embedding generated for the user node by the existing GNN model and (ii) the existing average local neighborhood user node embedding for the user node;

determining a second user value that is a dot product of: (i) an embedding generated for the user node by the updated GNN model and (ii) the new average local neighborhood user node embedding for the user node; and

determining a user node difference between the first user value and the second user value;

determining a user node average distance value that is an average of the user node difference determined in respect of the at least some of the user nodes;

for each of the at least some of the item nodes that are included in both the existing graph and the new graph:

determining a local neighborhood set of user nodes in the existing graph for the item node;

determining an existing average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the existing GNN model;

determining a new average local neighborhood item node embedding for the item node based on an average of embeddings generated for the user nodes in the neighborhood set by the updated GNN model;

determining a first item value that is a dot product of: (i) an embedding generated for the item node by the existing GNN model and (ii) the existing average local neighborhood item node embedding for the item node;

determining a second item value that is a dot product of: (i) an embedding generated for the item node by the updated GNN model and (ii) the new average local neighborhood item node embedding for the item node; and

determining an item node difference between the first item value and the second item value;

determining an item node average distance value that is an average of the item node difference determined in respect of the at least some of the item nodes;

wherein the local structure distillation component is based on a sum of the user node average distance value and the item node average distance value; and

replacing the existing GNN model of the RS with the updated GNN model.