Granted Patent B2

US 12,423,527 · App. 17/723,969 · Granted Sep 23, 2025

Inventor: Linfeng Song (Bellevue, WA)

Assignee: TENCENT AMERICA LLC

G06F40/35G06N3/047G06N3/088

Abstract

Claims (67)

1. A natural language processing method using abstract meaning representation (AMR) coreference resolution, the method performed by at least one processor and comprising:

receiving an input representation, wherein the input representation is comprised of an AMR graph;

encoding the input representation via a variational graph autoencoder (VGAE) pretrained using at least AMR graphs other than the AMR graph;

determining one or more concept identifiers from the encoded VGAE input representation; and

determining one or more coreference clusters from the determined concept identifiers; and

receiving a first set of an information loss related to the encoded input representation via the VGAE and based on embeddings for each AMR node, a plurality of character-level embeddings, one or more token-level embeddings, and a fixed embedding generated by a pretrained biderection encoder representations from transformers (BERT) model,

wherein the information loss depends on a Kullback-Leibler divergence,

wherein the first set of the information loss comprises an edge set loss value and a variational restriction on one or more hidden parameter values, and

wherein the information loss is L VGAE =L edge +L var =E q(Z|X,A) [log p(A′|Z)]−KL[q(Z|X, A)∥p(Z)], of which

A′ represents a value defined by an identity matrix,

X represents a variable of a Gaussian prior distribution of Z which indicates a stochastic latent variable,

KL(·∥·) represents the Kullback-Leibler divergence,

the edge set loss value is represented as L edge , and

the variational restrictions are represented as L var .

2. The method of claim 1 , further comprising:

determining one or more first embedding values for one or more nodes of the input representation.

3. The method of claim 2 , further comprising:

determining one or more second and third embedding values for the one or more nodes of the input representation.

4. The method of claim 1 , further comprising:

encoding one or more nodes of the input representation into a first representation having contextual information via a local graph encoder.

5. The method of claim 4 , further comprising:

selecting one or more hidden layers from the encoded one or more nodes.

6. The method of claim 5 , wherein the selected one or more hidden layers are at least partially modeled by a Gaussian distribution.

7. The method of claim 5 , further comprising:

decoding the one or more encoded nodes or the selected one or more hidden layers.

8. A computing device, comprising:

at least one memory configured to store computer program code; and

at least one processor configured to access the computer program code and operate as instructed by the computer program code, the computer program code comprising:

first receiving code configured to cause the at least one processor to receive an input representation, wherein the input representation is comprised of an AMR graph;

first encoding code configured to cause the at least one processor to encode the input representation via a variational graph autoencoder (VGAE) pretrained using at least AMR graphs other than the AMR graph;

first determining code configured to cause the at least one processor to determine one or more concept identifiers from the encoded VGAE input representation; and

second determining code configured to cause the at least one processor to determine one or more coreference clusters from the determined concept identifiers,

further receiving code configured to cause the at least one processor to receive a first set of an information loss related to the encoded input representation via the VGAE and based on embeddings for each AMR node, a plurality of character-level embeddings, one or more token-level embeddings, and a fixed embedding generated by a pretrained biderection encoder representations from transformers (BERT) model,

wherein the information loss depends on a Kullback-Leibler divergence,

wherein the first set of the information loss comprises an edge set loss value and a variational restriction on one or more hidden parameter values, and

wherein the information loss is L VGAE =L edge +L var =E q(Z|X,A) [log p(A′|Z)]−KL[q(Z|X, A)∥p(Z)], of which

A′ represents a value defined by an identity matrix,

X represents a variable of a Gaussian prior distribution of Z which indicates a stochastic latent variable,

KL (·∥·) represents the Kullback-Leibler divergence,

the edge set loss value is represented as L edge , and

the variational restrictions are represented as L var .

9. The computing device according to claim 8 , further comprising:

third determining code configured to cause the at least one processor to determine one or more first embedding values for one or more nodes of the input representation.

10. The computing device of claim 9 , further comprising:

third determining code configured to cause the at least one processor to determine one or more second and third embedding values for the one or more nodes of the input representation.

11. The computing device of claim 8 , wherein the first encoding code is further configured to cause the at least one processor to:

encode one or more nodes of the input representation into a first representation having contextual information via a local graph encoder.

12. The computing device of claim 11 , further comprising:

selecting code configured to cause the at least one processor to select one or more hidden layers from the encoded one or more nodes.

13. The computing device of claim 12 , wherein the selected one or more hidden layers are at least partially modeled by a Gaussian distribution.

14. The computing device of claim 13 , further comprising:

decoding code configured to cause the at least one processor to decode the one or more encoded nodes or the selected one or more hidden layers.

15. A non-transitory computer-readable storage medium storing program instructions that cause at least one processor to:

receive an input representation, wherein the input representation is comprised of an AMR graph;

encode, via a variational graph autoencoder (VGAE), the input representation pretrained using at least AMR graphs other than the AMR graph;

determine one or more concept identifiers to the encoded VGAE input representation;

determine one or more coreference clusters from the determined concept identifiers; and

receive a first set of an information loss related to the encoded input representation via the VGAE and based on embeddings for each AMR node, a plurality of character-level embeddings, one or more token-level embeddings, and a fixed embedding generated by a pretrained biderection encoder representations from transformers (BERT) model,

wherein the information loss depends on a Kullback-Leibler divergence,

wherein the first set of the information loss comprises an edge set loss value and a variational restriction on one or more hidden parameter values, and

wherein the information loss is L VGAE =L edge +L var =E q(Z|X,A) [log p(A′|Z)]−KL[q(Z|X, A)∥p(Z)], of which

A′ represents a value defined by an identity matrix,

X represents a variable of a Gaussian prior distribution of Z which indicates a stochastic latent variable,

KL (·∥·) represents the Kullback-Leibler divergence,

the edge set loss value is represented as L edge , and

the variational restrictions are represented as L var .

16. The non-transitory computer-readable storage medium according to claim 15 , wherein the program instructions further cause the at least one processor to determine one or more first embedding values for one or more nodes of the input representation.

Assignments (1)

Continuity (1)

References Cited (31)