Device and method with transformer model implementation
A device and method with transformer model implementation are provided. The electronic device includes a processor configured to perform an inference by implementing a transformer model including a plurality of encoders and a plurality of decoders, and a memory configured to store instructions to be executed by the processor. Each of the encoders and the decoders includes an attention block that determines an attention value. The processor is configured to perform a first sub-softmax tile-wise operation in the attention block, perform a reduction operation to determine an adjustment factor based on a resulting value of the first sub-softmax operation, and perform a second sub-softmax tile-wise operation based on a resulting value of the reduction operation.
1 . An electronic device, comprising:
a processor configured to implement a transformer model for processing a sentence in natural language processing (NLP), the transformer model comprising a plurality of encoders and a plurality of decoders; and
a memory configured to store instructions to be executed by the processor,
wherein each of the plurality of encoders and the plurality of decoders comprise an attention block that is configured to determine an attention value corresponding to a similarity between words in an input sentence, and
wherein, when the instructions are executed by the processor, the processor is further configured to:
perform, in an inference operation a first sub-softmax tile-wise operation in the attention block, perform a reduction operation to determine an adjustment factor based on a resulting value of the first sub-softmax tile-wise operation, and perform a second sub-softmax tile-wise operation based on a resulting value of the reduction operation.
2 . The electronic device of claim 1 , wherein the processor is further configured to:
perform a first matrix operation based on a query and a key, and perform the first sub-softmax tile-wise operation based on a resulting value of the first matrix operation.
3 . The electronic device of claim 2 , wherein the processor is further configured to:
perform a scale and mask operation on the resulting value of the first matrix operation, and perform the first sub-softmax tile-wise operation based on a resulting value of the scale and mask operation.
4 . The electronic device of claim 1 , wherein the processor is further configured to:
perform the first sub-softmax tile-wise operation by detecting a first maximum value tile-wise from the input sentence, determining a first exponentiated value of a resulting value obtained by subtracting the first maximum value from each value of the input sentence, and determining a first sum value of the first exponentiated value tile-wise.
5 . The electronic device of claim 2 , wherein the first matrix operation and the first sub-softmax operation are same in a data reading order and a data reading unit.
6 . The electronic device of claim 4 , wherein the processor is further configured to:
perform the reduction operation by:
determining a second exponentiated value of a resulting value obtained by subtracting, from the first maximum value, a second maximum value which is a maximum value of all tiles of the data;
determining a second sum value by applying, to the first sum value, the second exponentiated value corresponding to each tile of the data; and
determining, to be the adjustment factor, a resulting value obtained by dividing the second exponentiated value by the second sum value.
7 . The electronic device of claim 6 , wherein the processor is further configured to:
perform the second sub-softmax operation by applying the adjustment factor to the first exponentiated value.
8 . The electronic device of claim 1 , wherein the processor is further configured to:
perform a second tile-wise matrix operation based on a resulting value of the second sub-softmax tile-wise operation and a given data value.
9 . The electronic device of claim 8 , wherein the second sub-softmax tile-wise operation and the second tile-wise matrix operation are same in a data reading order and a data reading unit.
10 . An electronic device, comprising:
a processor configured to implement a transformer model for processing a sentence in natural language processing (NLP), the transformer model comprising a plurality of encoders and a plurality of decoders;
a memory configured to store instructions to be executed by the processor; and
reduction circuitry,
wherein each of the plurality of encoders and the plurality of decoders comprise an attention block that is configured to determine an attention value corresponding to a similarity between words in an input sentence,
wherein, when the instructions are executed by the processor, the processor is further configured to perform a first sub-softmax tile-wise operation in the attention block,
wherein the reduction circuitry is configured to perform a reduction operation to determine an adjustment factor based on a resulting value of the first sub-softmax tile-wise operation, and
wherein the processor is further configured to perform a second sub-softmax tile-wise operation based on a resulting value of the reduction operation.
11 . The electronic device of claim 10 , wherein the processor is configured to perform, in an inference operation, the first sub-softmax tile-wise operation by detecting a first maximum value tile-wise from the input sentence, determining a first exponentiated value of a resulting value obtained by subtracting the first maximum value from each value of the input sentence, and determining a first sum value of the first exponentiated value tile-wise.
12 . The electronic device of claim 11 , wherein the reduction circuitry comprises a comparator, an adder-subtractor, an exponent processor, a multiplication and division operator, and another memory.
13 . The electronic device of claim 12 , wherein the reduction circuitry is configured to:
read the first maximum value and the first sum value from the other memory;
detect, by the comparator, a second maximum value which is a maximum value of all tiles of the input sentence;
determine, by the adder-subtractor, a resulting value obtained by subtracting, from the first maximum value, a second maximum value detected row-wise from the input sentence;
determine, by the exponent processor, a second exponentiated value of a resulting value obtained by subtracting the second maximum value from the first maximum value; and
determine, by the multiplication and division operator, a second sum value by applying the second exponentiated value corresponding to each tile of the input sentence to the first sum value and determine, to be the adjustment factor, a resulting value obtained by dividing the second exponentiated value by the second sum value.
14 . The electronic device of claim 13 , wherein the processor is further configured to:
perform the second sub-softmax operation by applying the adjustment factor to the first exponentiated value.
15 . A processor-implemented method, comprising:
determining, by a processor, resulting data from input sentence by performing an inference on the input sentence by implementing a transformer model for processing a sentence in natural language processing (NLP), the transformer model comprising a plurality of encoders and a plurality of decoders,
wherein the determining of the resulting data by the processor comprises:
determining an attention value corresponding to a similarity between words in the input sentence, from each of the plurality of encoders and the plurality of decoders,
wherein the determining of the attention value comprises:
performing a first sub-softmax tile-wise operation;
performing a reduction operation to determine an adjustment factor based on a resulting value of the first sub-softmax tile-wise operation; and
performing a second sub-softmax tile-wise operation based on a resulting value of the reduction operation.
16 . The method of claim 15 , wherein the performing of the first sub-softmax tile-wise operation comprises:
detecting a first maximum value tile-wise from the input sentence;
determining a first exponentiated value of a resulting value obtained by subtracting the first maximum value from each value of the input sentence; and
determining a first sum value of the first exponentiated value tile-wise.
17 . The method of claim 16 , wherein the performing of the reduction operation comprises:
determining a second exponentiated value of a resulting value obtained by subtracting, from the first maximum value, a second maximum value which is a maximum value of all tiles of the input sentence;
determining a second sum value by applying, to the first sum value, the second exponentiated value corresponding to each tile of the input sentence; and
determining, to be the adjustment factor, a resulting value obtained by dividing the second exponentiated value by the second sum value.
18 . The method of claim 17 , wherein the performing of the second sub-softmax operation comprises:
applying the adjustment factor to the first exponentiated value.
19 . The method of claim 15 , wherein the reduction operation is performed by a reduction operator comprising a comparator, an adder-subtractor, an exponent processor, a multiplication and division operator, and a memory.
20 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the inference method of claim 15 .