Learning apparatus, text generation apparatus, learning method, text generation method and program
A learning device includes a memory; and a processor configured to execute answer generation means for taking data including text, and a question text related to the data as inputs; creating, by using a model parameter of a neural network, a token sequence that takes visual information in the data into consideration, and generating an answer text to the question text, based on the created token sequence; and learning means for learning the model parameter by using the answer text and a correct answer text to the question text.
1 . A learning apparatus, comprising:
a memory; and
a processor configured to execute:
receiving, as inputs, image data of an image and a question text related to the image data, wherein the image depicts a rendered text in a region of the image, the image data comprises visual content and the rendered text as a part of the visual content, and the rendered text comprises at least a part of a word and visual information of the at least the part of the word;
creating a token sequence comprising query token sequence and image data token sequence, wherein the query token sequence corresponds to the question text, and the image data token sequence is generated from both a text token and an image token, the text token is generated based at least on automatically recognizing the rendered text in the region of the image, the image token is generated based at least on image features of the visual content in the region of the image, and the creating the token sequence further comprises encoding the text token according to the visual information in the region of the image by performing language-with-visual-effect understanding operations;
generating, as output, an answer text to the question text, based on the created token sequence and a model parameter, thereby the answer text is according to at least the visual information effect of the rendered text; and
learning the model parameter by using the answer text and a correct answer text to the question text.
2 . The learning apparatus according to claim 1 ,
wherein, the visual information comprises at least either one of a location of the rendered text in the image data or a size of the rendered text, and
wherein the processor is configured to further execute: extracting a region based on a visual feature from the image data, and creating the token sequence, based on the question text, text included in the region, and information related to the region.
3 . The learning apparatus according to claim 2 , wherein the processor is further configured to execute calculating a probability that the region is necessary to generate the answer text,
wherein the generating further comprises generating the answer text, based on a sequence obtained by transforming the token sequence by using the probability calculated by the calculating, and
the learning learns the model parameter by using the probability calculated by the calculating, and a loss from correct information indicating whether or not the region is necessary to generate the answer text.
4 . The learning apparatus according to claim 1 , the processor configured to further execute:
extracting a plurality of regions from the image data; and
for each region in the plurality of regions:
generating one or more text tokens from text in a region of the plurality of regions; and
generating image token from visual feature of the region of the plurality of regions.
5 . The learning apparatus according to claim 4 , wherein the text is obtained by performing an optical character recognition of the region of the plurality of regions.
6 . The learning apparatus according to claim 4 , wherein the plurality of regions is extracted based on visual features of the visual content.
7 . The learning apparatus according to claim 4 , wherein the plurality of regions comprises text-containing regions and non-text regions.
8 . The learning apparatus according to claim 4 , wherein image data tokens in the image data token sequence comprise visual feature values.
9 . The learning apparatus according to claim 1 , wherein the image data comprises an integrated visual content in which text and non-text elements coexist within a unified visual representation.
10 . The learning apparatus according to claim 1 ,
wherein the image data token sequence is generated from both text token corresponding to the text included in the visual content and image token corresponding to the image features of the visual content.
11 . The learning apparatus according to claim 1 ,
wherein the processor is further configured to execute:
generating one or more text tokens and one or more image tokens from each of a plurality of extracted regions, wherein the one or more text tokens correspond to the text included in the visual content and the one or more image tokens correspond to the image features of the visual content.
12 . The learning apparatus according to claim 1 , the processor further configured to execute:
extract one or more regions from the image data; and
generate at least either one of the text token or the image token depending on content of the extracted one or more regions.
13 . A text generation apparatus, comprising:
a memory; and
a processor configured to execute:
receiving, as inputs, image data of an image and a question text related to the image data, wherein the image depicts a rendered text in a region of the image, the image data comprises visual content and the rendered text as a part of the visual content, and the rendered text comprises at least a part of a word and visual information of the at least the part of the word;
creating, by using a learned model parameter of a neural network, a token sequence comprising query token sequence and image data token sequence, wherein the query token sequence corresponds to the question text, the image data token sequence is generated from both a text token and an image token, the text token is generated based at least on automatically recognizing the rendered text in the region of the image, the image token is generated based at least on image features of the visual content in the region of the image, and the creating the token sequence further comprises encoding the text token according to the visual information in the region of the image by performing language-with-visual-effect understanding operations;
generating, as output, an answer text to the question text, based on the created token sequence and the learned model parameter, thereby the answer text is according to at least the visual information effect of the rendered text.
14 . The text generation apparatus according to claim 13 ,
wherein, the visual information comprises at least either one of a location of the rendered text in the image data or a size of the rendered text, and
wherein the creating extracts a region based on a visual feature from the image data, and creates the token sequence, based on the question text, text included in the region, and information related to the region.
15 . The text generation apparatus according to claim 14 , wherein the information related to the region includes information indicating a location of the region in the image data, and information representing a feature of the region.
16 . The text generation apparatus according to claim 14 , wherein the processor is further configured to execute calculating a probability that the region is necessary to generate the answer text, and
wherein creating generates the answer text, based on a sequence obtained by transforming the token sequence by using the probability calculated by the calculating.
17 . A learning method executed by a computer including a memory; and a processor, the method comprising:
receiving, as inputs, image data of an image and a question text related to the image data, wherein the image depicts a rendered text in a region of the image, the image data comprises visual content and the rendered text as a part of the visual content, and the rendered text comprises at least a part of a word and visual information of the at least the part of the word;
creating, by using a model parameter of a neural network, a token sequence comprising query token sequence and image data token sequence, wherein the query token sequence corresponds to the question text, and the image data token sequence is generated from both a text token and an image token, the text token is generated based at least on automatically recognizing the rendered text in the region of the image, the image token is generated based at least on image features of the visual content in the region of the image, and the creating the token sequence further comprises encoding the text token according to the visual information in the region of the image by performing language-with-visual-effect understanding operations;
generating, as output, an answer text to the question text, based on the created token sequence and the model parameter, thereby the answer text is according to at least the visual information effect of the rendered text; and
learning the model parameter by using the answer text and a correct answer text to the question text.
18 . The learning method according to claim 17 ,
wherein the image data token sequence is generated from both text token corresponding to the text included in the visual content and image token corresponding to the image features of the visual content.
19 . The learning method according to claim 17 ,
wherein, the visual information comprises at least either one of a location of the rendered text in the image data or a size of the rendered text, and
wherein the processor is further configured to execute:
generating one or more text tokens and one or more image tokens from each of a plurality of extracted regions, wherein the one or more text tokens correspond to the text included in the visual content and the one or more image tokens correspond to the image features of the visual content.
20 . A non-transitory computer-readable recording medium having computer-readable instructions stored thereon, which when executed, cause a computer including a memory and a processor to function as the learning apparatus according to claim 1 .