Apparatus and method for generating text from image and method of training model for generating text from image
An apparatus for generating text from an image may comprise: a memory configured to store at least one instruction; and a processor configured to execute the at least one instruction, wherein the processor is further configured to generate encoding information for an image based on the image and extract text information related to content of the image based on a degree of association with the encoding information.
1 . An apparatus for generating text from an image, the apparatus comprising:
a memory configured to store at least one instruction; and
a processor configured to execute the at least one instruction,
wherein the processor is further configured to:
convert the image into an image token sequence using a learned codebook;
generate encoding information comprising the image token sequence; and
generate text information describing semantic content of the image by applying an image-to-text translation model that receives the encoding information as an input sequence, and
wherein the image-to-text translation model is configured to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
2 . The apparatus of claim 1 , wherein each token of the image token sequence corresponds to a unit block of the image, the unit block having a size of N pixels×M pixels, N and M being positive integers greater than one.
3 . The apparatus of claim 2 , wherein N and M are equivalent.
4 . The apparatus of claim 1 , wherein the image-to-text translation model comprises a transformer-based encoder-decoder architecture configured to generate the text information based on the learned degrees of association between the image tokens and the text tokens.
5 . The apparatus of claim 4 , wherein
at least one encoder layer of the transformer-based encoder-decoder architecture comprises a self-attention neural network layer and a feed-forward neural network layer, and
at least one decoder layer of the transformer-based encoder-decoder architecture comprises a self-attention neural network layer, an encoder-decoder attention neural network layer, and a feed-forward neural network layer.
6 . The apparatus of claim 1 , wherein the processor is further configured to generate image-text synthetic data for the image by combining the image and the text information.
7 . The apparatus of claim 6 , wherein the processor is further configured to generate an image based on the text using the image-text synthetic data.
8 . A method of generating text from an image performed by a processor executing at least one instruction stored in a memory, the method comprising:
converting the image into an image token sequence using a learned codebook;
generating encoding information comprising the image token sequence; and
generating text information describing semantic content of the image by applying an image-to-text translation model that receives the encoding information as an input sequence,
wherein the image-to-text translation model is configured to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
9 . The method of claim 8 , wherein each token of the image token sequence corresponds to a unit block of the image, the unit block having a size of N pixels×M pixels, N and M being positive integers greater than one.
10 . The method of claim 9 , wherein N and M are equivalent.
11 . The method of claim 8 , wherein the image-to-text translation model comprises a transformer-based encoder-decoder architecture configured to generate the text information based on the learned degrees of association between the image tokens and the text tokens.
12 . The method of claim 11 , wherein
at least one encoder layer of the transformer-based encoder-decoder architecture comprises a self-attention neural network layer and a feed-forward neural network layer, and
at least one decoder layer of the transformer-based encoder-decoder architecture comprises a self-attention neural network layer, an encoder-decoder attention neural network layer, and a feed-forward neural network layer.
13 . The method of claim 8 , further comprising combining the image and the text information to generate image-text synthetic data for the image.
14 . The method of claim 13 , further comprising generating an image based on the text using the image-text synthetic data.
15 . A method of training an image-to-text translation model for generating text describing semantic content of an image, performed by a processor executing at least one instruction stored in a memory, the method comprising:
converting an image into an image token sequence using a learned codebook;
generating encoding information comprising the image token sequence; and
training the image-to-text translation model to learn a function for generating text information describing semantic content of the image using the encoding information as an input sequence,
wherein the function enables the image-to-text translation model to generate the text information based on learned degrees of association between image tokens in the encoding information and text tokens.
16 . The method of claim 15 , wherein each token of the image token sequence corresponds to a unit block of the image, the unit block having a size of N pixels×M pixels, N and M being positive integers greater than one.
17 . The method of claim 15 , further comprising:
inputting a training image to an encoder-decoder model; and
training the encoder-decoder model to learn a function for generating the learned codebook from the training image.
18 . The method of claim 15 , wherein the image-to-text translation model comprises a transformer-based encoder-decoder architecture configured to generate, using the function, the text information based on the learned degrees of association between the image tokens and the text tokens.
19 . The method of claim 15 , further comprising combining the image and the text information to generate image-text synthetic data for the image.
20 . The method of claim 19 , further comprising training a text-based image generation model to learn a function for generating an image based on text using the image-text synthetic data.