Systems for context-aware named entity recognition
To perform a Named Entity Recognition (NER) operation that accounts for semantic relationships between entity and document text while remaining scalable, entity text and text from portions of documents are encoded separately, in some cases using different encoders such as when the document text also includes layout information. A cross-attention module is then used to determine a weighted embedding for each document embedding, based on the document embedding and each entity embedding. The weighted embeddings include weights indicative of semantic relationships between the document text and the text of each entity. Correspondence between the weighted embeddings and the corresponding document embedding is then used to determine scores representing the probability that an entity is referenced in a portion of a document. Because the weighted embeddings are used, these scores account for semantic relationships between entity and document text.
1 . A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
encode, using a first encoder, entity data representing a first named entity to generate a first entity embedding and a second entity embedding;
encode, using a second encoder, first document data representing a first portion of a document to generate a first document embedding
and second document data representing a second portion of the document to generate a second document embedding;
determine a first weighted embedding based on:
a cross-attention function in which the first document embedding is used as a query for the cross-attention function and the first entity embedding and the second entity embedding are used as keys and values for the cross-attention function,
wherein the first weighted embedding includes: one or more first values representing first semantic relationships between the first document data, the first entity embedding, and the second entity embedding;
determine a second weighted embedding based on:
the cross-attention function in which the second document embedding is used as a query for the cross-attention function and the first entity embedding and the second entity embedding are used as keys and values for the cross-attention function,
wherein the second weighted embedding includes: one or more second values indicative of second semantic relationships between the second document data, the first entity embedding, and the second entity embedding;
determine a first score based on correspondence between the first weighted embedding and the first document embedding, wherein the first score represents occurrences, within the first document data, of text associated with the entity data;
determine a second score based on correspondence between the second weighted embedding and the second document embedding, wherein the second score represents occurrences, within the second document data, of text associated with the entity data; and
generate output based on the first score and the second score, wherein the output is indicative of inclusion of text that references the first named entity within one or more of the first document data or the second document data.
2 . The system of claim 1 , wherein:
the entity data comprises text data, and
the first encoder is a first type of encoder configured to determine embeddings based on text data,
at least one of the first document data or the second document data comprises text data and layout data; and
the second encoder is a second type of encoder that differs from the first type of encoder and is configured to determine embeddings based on text data and layout data.
3 . The system of claim 1 , further comprising computer-executable instructions to:
determine a first matrix (T) based on the first document embedding and the second document embedding; and
determine a second matrix (L) based on the first entity embedding and the second entity embedding;
wherein the cross-attention function is used to determine a third matrix (U) based on the equation: U=softmax (TL T )L; and
wherein the first weighted embedding and the second weighted embedding are determined based on the third matrix.
4 . A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
determine a first embedding based on first text;
determine a second embedding based on second text;
determine a third embedding based on third text;
determine a fourth embedding based on fourth text;
determine a fifth embedding and a sixth embedding based on a cross-attention function in which the third embedding and fourth embedding are inputs used as queries for the cross-attention function and the first embedding and the second embedding are inputs used as one or more of keys or values for the cross-attention function,
wherein the fifth embedding includes one or more first values indicative of first semantic relationships between the first text, the second text, and the third text, and
wherein the sixth embedding includes one or more second values indicative of second semantic relationships between the first text, the second text, and the fourth text;
determine a first score based on correspondence between the fifth embedding and the third embedding;
determine a second score based on correspondence between the sixth embedding and the fourth embedding; and
generate output based on the first score and the second score.
5 . The system of claim 4 , further comprising computer-executable instructions to:
encode first data using a first encoder to determine the first embedding;
encode second data using the first encoder to determine the second embedding;
encode third data using a second encoder that differs from the first encoder to determine the third embedding; and
encode fourth data using the second encoder to determine the fourth embedding.
6 . The system of claim 5 , wherein:
the first data comprises the first text;
the second data comprises the second text;
the third data comprises the third text and first layout data;
the fourth data comprises the fourth text and second layout data; and
the second encoder is configured to determine embeddings based at least in part on layout data.
7 . The system of claim 4 , wherein:
the first score is further based at least in part on a first dot product associated with the fifth embedding and the third embedding; and
the second score is further based at least in part on a second dot product associated with the sixth embedding and the fourth embedding.
8 . The system of claim 4 , further comprising computer-executable instructions to:
determine a first matrix (T) based on the third embedding and the fourth embedding; and
determine a second matrix (L) based on the first embedding and the second embedding;
wherein the cross-attention function is used to determine a third matrix (U) that comprises the fifth embedding and the sixth embedding based on the first matrix and the second matrix; and
wherein the fifth embedding and the sixth embedding are determined based on the third matrix.
9 . The system of claim 8 , wherein the cross-attention function is used to
determine the third matrix based on the equation: U=softmax (TL T )L.
10 . The system of claim 4 , wherein:
the first score represents a first probability that the third text associated with the third embedding is associated with one or more of the first text associated with the first embedding or the second text associated with the second embedding; and
the second score represents a second probability that the fourth text associated with the fourth embedding is associated with the one or more of the first text associated with the first embedding or the second text associated with the second embedding.
11 . The system of claim 4 , wherein:
the first text and the second text are associated with a first entity;
the third text is associated with a first portion of a document;
the fourth text is associated with a second portion of the document;
the first score is indicative of inclusion or exclusion of text associated with the first entity within the first portion of the document; and
the second score is indicative of inclusion or exclusion of text associated with the first entity within the second portion of the document.
12 . A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
determine a first embedding based on first text;
determine a second embedding based on second text;
determine a third embedding based on third text;
determine a fourth embedding based on a cross-attention function, the first embedding and the second embedding, wherein the fourth embedding includes one or more first values indicative of semantic relationships between the first text and the second text;
determine a fifth embedding based on the cross-attention function, the first embedding, and the third embedding, wherein the fifth embedding includes one or more second values indicative of semantic relationships between the first text and the third text;
determine a first score based on correspondence between the fourth embedding and the first embedding, wherein the first score represents occurrences, within the first text, of text associated with the second text;
determine a second score based on correspondence between the fifth embedding and the first embedding, wherein the second score represents occurrences, within the second text, of text associated with the third text; and
generate output based on the first score and the second score.
13 . The system of claim 12 , further comprising computer-executable instructions to:
determine a sixth embedding based on fourth text;
determine a seventh embedding based on the cross-attention function, the sixth embedding and the second embedding, wherein the seventh embedding includes one or more third values indicative of semantic relationships between the fourth text and the second text;
determine an eighth embedding based on the cross-attention function, the sixth embedding, and the third embedding, wherein the eighth embedding includes one or more fourth values indicative of semantic relationships between the fourth text and the third text;
determine a third score based on correspondence between the seventh embedding and the sixth embedding; and
determine a fourth score based on correspondence between the eighth embedding and the sixth embedding;
wherein the output is further based on the third score and the fourth score.
14 . The system of claim 13 , wherein the first embedding and the sixth embedding are inputs to the cross-attention function used as queries, and the second embedding and the third embedding are inputs to the cross-attention function used as one or more of keys or values.
15 . The system of claim 13 , further comprising computer-executable instructions to:
determine a first matrix (T) based on the first embedding; and
determine a second matrix (L) based on the second embedding;
wherein the cross-attention function is used to determine a third matrix (U) that comprises the fourth embedding based on the first matrix and the second matrix.
16 . The system of claim 15 , wherein the cross-attention function is used to determine the third matrix based on the equation: U=softmax (TL T )L.
17 . The system of claim 12 , further comprising computer-executable instructions to:
encode the second text to determine the second embedding using a first encoder;
encode the third text to determine the third embedding using the first encoder; and
encode the first text and layout data associated with the first text to determine the first embedding using a second encoder, wherein the second encoder differs from the first encoder and is configured to determine embeddings based in part on layout data.
18 . The system of claim 12 , wherein the first score is further based at least in part on a dot product associated with the fourth embedding and the first embedding, and the second score is further based at least in part on a dot product associated with the fifth embedding and the first embedding.
19 . The system of claim 12 , wherein the first score represents a probability that the first text is associated with the second text, and the second score represents a probability that the first text is associated with the third text.
20 . The system of claim 12 , wherein:
the first text is associated with at least a portion of a document;
the second text is associated with a first entity;
the third text is associated with a second entity;
the first score is indicative of inclusion or exclusion of text associated with the first entity within the at least a portion of the document; and
the second score is indicative of inclusion or exclusion of text associated with the second entity within the at least a portion of the document.