Granted Patent B2

US 12,277,402 · App. 18/453,427 · Granted Apr 15, 2025

Inventors: Thomas Joachim Zenkel (Bamberg, DE); Joern Wuebker (Berlin, DE); John Sturdy DeNero (Berkeley, CA)

Assignee: Lilt, Inc.

G06F40/58G06F40/284G06N3/045

Abstract

Claims (88)

1. A computer-implemented method, comprising:

storing, in computer main memory, an encoding comprising a tokenized vocabulary of a source language and a target language;

training, using the encoding, a forward neural model programmed for translating from the source language to the target language, a forward alignment layer that is programmatically coupled to the forward neural model, a backward neural model for translating from the target language to the source language, and a backward alignment layer that is programmatically coupled to the backward neural model,

storing, in computer main memory, a pairing comprising a source representation of a source sentence associated with the source language and a corresponding target representation of a target sentence associated with the target language;

extracting, based on the source representation and the target representation, forward attention logits from the forward alignment layer and backward attention logits from the backward alignment layer;

programmatically inferring a symmetrized attention matrix that jointly optimizes the likelihood of the pairing under the forward neural model and the backward neural model; and

generating and digitally storing, based on the symmetrized attention matrix, a plurality of first hard alignments between source words comprising source tokens in the source sentence and target words comprising target tokens in the target sentence.

2. The method of claim 1 , at least one of the forward neural model and the backward neural model comprising a Transformer submodel comprising a 6-layer encoder, a 3-layer decoder, 8 attention heads per layer, 256-dimensional hidden states, and 256-dimensional embedding vectors; each alignment layer comprising 256-dimensional hidden states, 256-dimensional embedding vectors, and one attention head; and the Transformer submodel and each alignment layer using a dropout rate of 0.1.

3. The method of claim 1 , further comprising generating the first hard alignments by generating one or more alignment links that programmatically associate one or more source tokens with one or more target tokens, and generating a first hard alignment between each source word and each target word respectively comprising a source token and a target token programmatically associated by an alignment link.

4. The method of claim 3 , further comprising:

projecting, based on the generated first hard alignments, one or more tags from the source sentence to the target sentence, including at least one of underlining, an HTML tag, an XML tag, an SGML tag, a LaTeX tag, italicization, a sentiment annotation, an entity annotation, an intent annotation, a relationship annotation, highlighting, coloring, bolding, superscript, subscript, strikethrough, font, a comment, a footnote, or a link; and

storing, in computer main memory, the target sentence with the projected tags.

5. The method of claim 4 , further comprising projecting the one or more tags based on the generated first hard alignments by executing:

determining, for each opening tag corresponding to a closing tag in the source sentence, a corresponding source tag span;

calculating, for each corresponding source tag span, an alignment score for every possible target tag span; and

projecting, into the target sentence, from each corresponding source tag span to a possible target tag span with the alignment score having the highest value, the opening and closing tags.

6. The method of claim 1 , the first hard alignments being generated by:

transforming the attention logits from the symmetrized attention matrix into forward and backward attention probabilities using a softmax( ) function;

merging the forward and backward attention probabilities into attention scores using a Hadamard product; and

selecting min(n, m) alignments that have the highest values in the merged attention scores, wherein n is a number of source tokens and m is a number of target tokens.

7. The method of claim 1 , further comprising:

programmatically uncoupling the forward alignment layer and the backward alignment layer from the forward neural model and the backward neural model respectively;

programmatically coupling a guided alignment layer comprising an unmasked self-attention sublayer to the forward neural model and the backward neural model;

training, with a plurality of hard alignments that includes the first hard alignments, using the encoding, the guided alignment layer;

storing, in computer main memory, a second pairing comprising a second source representation of a second source sentence associated with the source language and a corresponding second target representation of a second target sentence associated with the target language;

extracting, based on the second source representation and the second target representation, guided attention logits from the guided alignment layer; and

generating, based on the extracted guided attention logits, guided hard alignments between second source words comprising second source tokens in the second source sentence and second target words comprising second target tokens in the second target sentence.

8. The method of claim 7 , each of the forward neural model and the backward neural model comprising a Transformer submodel comprising a 6-layer encoder, a 3-layer decoder, 8 attention heads per layer, 256-dimensional hidden states, and 256-dimensional embedding vectors;

the forward and backward alignment layer each comprising 256-dimensional hidden states, 256-dimensional embedding vectors, and one attention head; and

the Transformer submodel and each of the forward and backward alignment layer using a dropout rate of 0.1.

9. The method of claim 7 , further comprising generating the guided hard alignments by generating one or more alignment links that programmatically associate one or more second source tokens with one or more second target tokens, and

generating a guided hard alignment between each second source word and each second target word respectively comprising a second source token and a second target token programmatically associated by an alignment link.

10. The method of claim 9 , further comprising:

projecting, based on the generated guided hard alignments, one or more tags from the second source sentence to the second target sentence, including at least one of underlining, an HTML tag, an XML tag, an SGML tag, a LaTeX tag, italicization, a sentiment annotation, an entity annotation, an intent annotation, a relationship annotation, highlighting, coloring, bolding, superscript, subscript, strikethrough, font, a comment, a footnote, or a link; and

storing, in computer main memory, the second target sentence with the projected tags.

11. The method of claim 10 , further comprising projecting the one or more tags based on the generated guided hard alignments by executing:

determining, for each opening tag corresponding to a closing tag in the second source sentence, a corresponding source tag span;

calculating, for each corresponding source tag span, an alignment score for every possible target tag span; and

projecting, into the second target sentence, from each corresponding source tag span to a possible target tag span with the alignment score having the highest value, the opening and closing tags.

12. The method of claim 1 , further comprising:

programmatically uncoupling the forward alignment layer and the backward alignment layer from the forward neural model and the backward neural model respectively;

programmatically coupling a guided alignment layer comprising an unmasked self-attention sublayer to the forward neural model;

training, with a plurality of hard alignments that includes the first hard alignments, using the encoding, the guided alignment layer;

storing, in computer main memory, a second pairing comprising a second source representation of a second source sentence associated with the source language and a corresponding second target representation of a second target sentence associated with the target language;

extracting, based on the second source representation and the second target representation, guided attention logits from the guided alignment layer; and

generating, based on the extracted guided attention logits, guided hard alignments between second source words comprising second source tokens in the second source sentence and second target words comprising second target tokens in the second target sentence.

13. The method of claim 12 , each of the forward neural model and the backward neural model comprising a Transformer submodel comprising a 6-layer encoder, a 3-layer decoder, 8 attention heads per layer, 256-dimensional hidden states, and 256-dimensional embedding vectors;

the forward and backward alignment layer each comprising 256-dimensional hidden states, 256-dimensional embedding vectors, and one attention head; and

each Transformer submodel and each of the forward and backward alignment layer using a dropout rate of 0.1.

14. The method of claim 12 , further comprising generating the guided hard alignments by generating one or more alignment links that programmatically associate one or more second source tokens with one or more second target tokens, and

generating a guided hard alignment between each second source word and each second target word respectively comprising a second source token and a second target token programmatically associated by an alignment link.

15. The method of claim 14 , further comprising:

projecting, based on the generated guided hard alignments, one or more tags from the second source sentence to the second target sentence, including at least one of underlining, an HTML tag, an XML tag, an SGML tag, a LaTeX tag, italicization, a sentiment annotation, an entity annotation, an intent annotation, a relationship annotation, highlighting, coloring, bolding, superscript, subscript, strikethrough, font, a comment, a footnote, or a link; and

storing, in computer main memory, the second target sentence with the projected tags.

16. The method of claim 15 , further comprising projecting the one or more tags based on the generated guided hard alignments by executing:

determining, for each opening tag corresponding to a closing tag in the second source sentence, a corresponding source tag span;

calculating, for each corresponding source tag span, an alignment score for every possible target tag span; and

projecting, into the second target sentence, from each corresponding source tag span to a possible target tag span with the alignment score having the highest value, the opening and closing tags.

17. A non-transitory computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

storing, in computer main memory, an encoding comprising a tokenized vocabulary of a source language and a target language;

training, using the encoding, a forward neural model programmed for translating from the source language to the target language, a forward alignment layer that is programmatically coupled to the forward neural model, a backward neural model for translating from the target language to the source language, and a backward alignment layer that is programmatically coupled to the backward neural model,

storing, in computer main memory, a pairing comprising a source representation of a source sentence associated with the source language and a corresponding target representation of a target sentence associated with the target language;

extracting, based on the source representation and the target representation, forward attention logits from the forward alignment layer and backward attention logits from the backward alignment layer;

programmatically inferring a symmetrized attention matrix that jointly optimizes the likelihood of the pairing under the forward neural model and the backward neural model; and

generating and digitally storing, based on the symmetrized attention matrix, a plurality of first hard alignments between source words comprising source tokens in the source sentence and target words comprising target tokens in the target sentence.

18. The non-transitory computer-readable storage medium of claim 17 , the steps further comprising:

programmatically uncoupling the forward alignment layer and the backward alignment layer from the forward neural model and the backward neural model respectively;

programmatically coupling a guided alignment layer comprising an unmasked self-attention sublayer to the forward neural model and the backward neural model;

training, with a plurality of hard alignments that includes the first hard alignments, using the encoding, the guided alignment layer;

storing, in computer main memory, a second pairing comprising a second source representation of a second source sentence associated with the source language and a corresponding second target representation of a second target sentence associated with the target language;

extracting, based on the second source representation and the second target representation, guided attention logits from the guided alignment layer; and

generating, based on the extracted guided attention logits, guided hard alignments between second source words comprising second source tokens in the second source sentence and second target words comprising second target tokens in the second target sentence.

19. A computer system, comprising:

one or more processors;

a memory comprising a set of instructions which when executed causes the one or more processors to execute a method, the method comprising:

storing, in computer main memory, an encoding comprising a tokenized vocabulary of a source language and a target language;

training, using the encoding, a forward neural model programmed for translating from the source language to the target language, a forward alignment layer that is programmatically coupled to the forward neural model, a backward neural model programmed for translating from the target language to the source language, and a backward alignment layer that is programmatically coupled to the backward neural model,

storing, in computer main memory, a pairing comprising a source representation of a source sentence associated with the source language and a corresponding target representation of a target sentence associated with the target language;

extracting, based on the source representation and the target representation, forward attention logits from the forward alignment layer and backward attention logits from the backward alignment layer;

programmatically inferring a symmetrized attention matrix that jointly optimizes the likelihood of the pairing under the forward neural model and the backward neural model; and

generating and digitally storing, based on the symmetrized attention matrix, a plurality of first hard alignments between source words comprising source tokens in the source sentence and target words comprising target tokens in the target sentence.

20. The computer system of claim 19 , the method further comprising:

programmatically uncoupling the forward alignment layer and the backward alignment layer from the forward neural model and the backward neural model respectively;

programmatically coupling a guided alignment layer comprising an unmasked self-attention sublayer to the forward neural model and the backward neural model;

training, with a plurality of hard alignments that includes the first hard alignments, using the encoding, the guided alignment layer;

storing, in computer main memory, a second pairing comprising a second source representation of a second source sentence associated with the source language and a corresponding second target representation of a second target sentence associated with the target language;

extracting, based on the second source representation and the second target representation, guided attention logits from the guided alignment layer; and

generating, based on the extracted guided attention logits, guided hard alignments between second source words comprising second source tokens in the second source sentence and second target words comprising second target tokens in the second target sentence.

Assignments (1)

Continuity (2)

References Cited (38)