Systems for determining representations of entities based on graph data using a language model

1 . A system comprising:

one or more memories storing computer-executable instructions; and

one or more hardware processors to execute the computer-executable instructions to:

determine entity data indicative of first text and a first image associated with a first entity;

determine correspondence between the entity data and graph data, wherein the graph data includes:

a plurality of nodes comprising at least a first node and a second node, wherein the first node represents first characteristics of the first entity and the second node represents second characteristics of a second entity; and

a plurality of edges comprising at least a first edge associated with the first node and the second node that represents a type of a relationship between the first entity and the second entity;

determine subgraph data that includes a portion of the graph data that is associated with the first entity, wherein the subgraph data includes at least the first node, the second node, and the first edge;

determine graph representation data based on the subgraph data and a graph encoder, wherein the graph representation data is indicative of the first characteristics, the type of relationship, and the second characteristics;

determine image representation data based on the first image and an image encoder, wherein the image representation data is indicative of third characteristics of the first image;

determine text representation data based on the first text and a text encoder, where in the text representation data is indicative of fourth characteristics of the first text;

determine input data based on the graph representation data, the image representation data, the text representation data, and input parameters associated with inputs to a large language model (LLM), wherein the input parameters define a threshold distance between the first entity and the second entity, and the graph representation data corresponds to the input parameters;

send the input data to the large language model (LLM) that is trained to determine representations based on inputs representing graph data, image data, and text data;

receive an output from the LLM; and

determine, based on the output from the LLM, an entity representation, wherein the entity representation is indicative of:

the first characteristics of the first entity indicated in the graph data,

the second characteristics of the second entity indicated in the graph data,

the third characteristics of the first image associated with the first entity, and

the fourth characteristics of the first text associated with the first entity.

2 . The system of claim 1 , further comprising computer-executable instructions to:

train the graph encoder to determine the graph representation data using training data comprising a plurality of nodes indicative of entity characteristics and a plurality of edges indicative of types of relationships between entities, wherein:

a first type of relationship between entities includes occurrence of a first user interaction associated with a third entity and a first session, and a second user interaction associated with a fourth entity and the first session;

the graph encoder is trained to predict occurrence of the first type of relationship; and

one or more second types of relationships that differ from the first type of relationship are represented in the graph representation data.

3 . The system of claim 1 , further comprising computer-executable instructions to:

train the large language model (LLM) to determine outputs based on inputs representing one or more of graph data, image data, or text data using training data that includes a plurality of pairs of values, a first loss function based on prediction of a first value of a pair of values based on a second value of the pair of values, and a second loss function based on contrastive loss associated with the first value and the second value.

4 . A system comprising:

one or more memories storing computer-executable instructions; and

one or more hardware processors to execute the computer-executable instructions to:

determine first entity data associated with a first entity, wherein the first entity data includes text data and image data associated with the first entity;

determine correspondence between the first entity data and graph data, wherein the graph data includes:

a plurality of nodes comprising at least a first node and a second node, wherein the first node represents first characteristics of the first entity and the second node represents second characteristics of a second entity; and

a plurality of edges comprising at least a first edge associated with the first node and the second node that represents a relationship between the first entity and the second entity;

determine first graph representation data based on at least a first portion of the graph data using a first machine learning model, wherein the first graph representation data is indicative of the first characteristics, the relationship, and the second characteristics;

determine first input data based on the first graph representation data, the text data, the image data, and one or more input parameters associated with inputs to a second machine learning model, wherein the one or more input parameters define a threshold distance between the first entity and the second entity, and the first graph representation data corresponds to one or more the input parameters;

send the first input data to the second machine learning model that is trained to determine representations based on inputs representing graph data, image data, and text data;

receive a first output from the second machine learning model; and

determine, based on the first output from the second machine learning model, a first entity representation indicative of at least a portion of the first characteristics of the first entity indicated in the graph data, the second characteristics of the second entity indicated in the graph data, the text data, and the image data.

5 . The system of claim 4 , further comprising computer-executable instructions to:

determine subgraph data based on the correspondence between the first entity data and the graph data and a threshold distance value indicative of a distance from the first node that represents the first entity, wherein the subgraph data comprises a first portion of the plurality of nodes and a second portion of the plurality of edges;

wherein the first graph representation data is further determined based on the subgraph data.

6 . The system of claim 4 , further comprising computer-executable instructions to:

determine text representation data based on at least a portion of the text data using a third machine learning model, wherein the text representation data is indicative of at least a portion of the text data and semantic information associated with the text data; and

wherein the first input data is further determined based in part on the text representation data.

7 . The system of claim 4 , further comprising computer-executable instructions to:

determine image representation data based on at least a portion of the image data using a third machine learning model, wherein the image representation data is indicative of one or more characteristics of the image data; and

wherein the first input data is further determined based in part on the image representation data.

8 . The system of claim 4 , further comprising computer-executable instructions to:

determine text representation data based on at least a portion of the text data using a third machine learning model, wherein the text representation data is indicative of at least a portion of the text data and semantic information associated with the text data; and

determine image representation data based on at least a portion of the image data using a fourth machine learning model, wherein the image representation data is indicative of one or more characteristics of the image data;

wherein the first input data is further determined based in part on the text representation data and the image representation data.

9 . The system of claim 4 , further comprising computer-executable instructions to:

train the first machine learning model to determine the first graph representation data using training data comprising a plurality of nodes indicative of characteristics of entities and a plurality of edges indicative of types of relationships between entities, wherein:

a first type of relationship between entities includes occurrence of a first user interaction associated with a third entity and a second user interaction associated with a fourth entity; and

the first machine learning model is trained to predict occurrence of the first type of relationship.

10 . The system of claim 4 , further comprising computer-executable instructions to:

train the second machine learning model to determine outputs based on inputs representing one or more of graph data, image data, or text data using training data that includes a plurality of pairs of values, a first loss function based on prediction of a first value of a pair of values based on a second value of the pair of values, and a second loss function based on contrastive loss associated with the first value and the second value.

11 . The system of claim 4 , wherein the first entity includes one of:

an item having one or more first item characteristics;

a search query including one or more second item characteristics;

a brand associated with one or more first items and one or more brand characteristics, wherein each first item of the one or more first items is associated with a respective one or more third item characteristics;

an item category associated with one or more second items and one or more category characteristics, wherein each second item of the one or more second items is associated with a respective one or more fourth item characteristics; or

a geographic region associated with one or more third items and one or more region characteristics, wherein each third item of the one or more third items is associated with a respective one or more fifth item characteristics.

12 . The system of claim 4 , further comprising computer-executable instructions to:

determine second entity data associated with a third entity;

determine correspondence between the second entity data and the graph data;

determine second graph representation data based on at least a second portion of the graph data using the first machine learning model;

determine second input data based at least in part on the second graph representation data;

provide the second input data to the second machine learning model;

determine, based on a second output from the second machine learning model, a second entity representation indicative of one or more third characteristics associated with the third entity;

determine output data associated with the first entity;

determine that the first entity representation corresponds to the second entity representation within a threshold similarity; and

include in the output data an indication associated with the third entity.

13 . A computer-implemented method comprising:

determining first entity data associated with a first entity, wherein the first entity data includes one or more of text data or image data associated with the first entity;

determining correspondence between the first entity data and graph data, wherein the graph data associates first characteristics of the first entity with second characteristics of a second entity and a type of relationship associated with the first entity and the second entity;

determining first graph representation data based on at least a first portion of the graph data and a first machine learning model, where in the first graph representation data is indicative of the first characteristics, the type of relationship, and the second characteristics;

determining first input data based at least in part on the first graph representation data, the one or more of the text data or the image data, and one or more input parameters associated with inputs to a second machine learning model, wherein the one or more input parameters define a threshold distance between the first entity and the second entity, and the first graph representation data corresponds to the one or more input parameters;

sending the first input data to the second machine learning model that is trained to determine representations based on inputs representing one or more of graph data, image data, or text data;

receiving a first output from the second machine learning model; and

determining, based on the first output from the second machine learning model, a first entity representation indicative of at least a portion of the first characteristics of the first entity indicated in the graph data, the second characteristics of the second entity indicated in the graph data, and the one or more of the text data or the image data.

14 . The method of claim 13 , wherein the graph data includes:

a plurality of nodes comprising at least a first node and a second node, wherein the first node represents the first characteristics of the first entity and the second node represents the second characteristics of the second entity; and

a plurality of edges comprising at least a first edge associated with the first node and the second node that represents the type of relationship between the first entity and the second entity.

15 . The method of claim 14 , further comprising:

determining subgraph data based on:

the correspondence between the first entity data and the graph data; and

a threshold distance value indicative of a distance from the first node;

wherein the subgraph data comprises a first portion of the plurality of nodes and a second portion of the plurality of edges, and the first graph representation data is further determined based on the subgraph data.

16 . The method of claim 13 , wherein the first entity data includes the text data associated with the first entity, the method further comprising:

determining text representation data based on at least a portion of the text data using a third machine learning model, wherein the text representation data is indicative of at least a portion of the text data and semantic information associated with the text data; and

wherein the first input data is further determined based in part on the text representation data.

17 . The method of claim 13 , wherein the first entity data includes the image data associated with the first entity, the method further comprising:

determining image representation data based on at least a portion of the image data using a third machine learning model, wherein the image representation data is indicative of one or more characteristics of the image data; and

wherein the first input data is further determined based in part on the image representation data.

18 . The method of claim 13 , wherein the first entity data includes the text data and the image data associated with the first entity, the method further comprising:

determining text representation data based on at least a portion of the text data using a third machine learning model, wherein the text representation data is indicative of at least a portion of the text data and semantic information associated with the text data; and

determining image representation data based on at least a portion of the image data using a fourth machine learning model, wherein the image representation data is indicative of one or more characteristics of the image data;

wherein the first input data is further determined based in part on the text representation data and the image representation data.

19 . The method of claim 18 , wherein the first machine learning model includes a graph encoder, the second machine learning model includes a large language model (LLM), the third machine learning model includes a text encoder, and the fourth machine learning model includes an image encoder.

20 . The method of claim 13 , further comprising:

training the first machine learning model to determine the first graph representation data using training data comprising a plurality of nodes indicative of characteristics of entities and a plurality of edges indicative of types of relationships between entities, wherein:

a first type of relationship between entities includes occurrence of a first user interaction associated with a third entity and a second user interaction associated with a fourth entity;

the first machine learning model is trained to predict occurrence of the first type of relationship; and

one or more second types of relationships that differ from the first type of relationship are represented in the first graph representation data.