Information processing apparatus, information processing method, learning method and moving object for predicting a region in an image corresponding to utterance
An information processing apparatus in embodiments performs at least one trained machine learning model that includes an encoder and a decoder. The encoder receives inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and outputs tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information. The decoder outputs a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.
1 . An information processing apparatus comprising
at least one processor configured to perform at least one trained machine learning model, wherein
the at least one trained machine learning model includes:
an encoder configured to receive inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and configured to output tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information; and
a decoder configured to output a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.
2 . The information processing apparatus according to claim 1 , wherein
the encoder includes a first encoder block and a second encoder block, the first encoder block being configured to output bi-modal features corresponding to the first image, the bi-modal features being obtained by fusing the language features of the text information into the visual features of the first image by use of an attention mechanism, the second encoder block being configured to output bi-modal features corresponding to the second image, the bi-modal features being obtained by fusing the language features of the text information into the visual features of the second image by use of an attention mechanism; and
the encoder either concatenates or adds the bi-modal features corresponding to the first image and the bi-modal features corresponding to the second image, and outputs the tri-modal features.
3 . The information processing apparatus according to claim 2 , wherein the encoder further outputs a visual feature map in which the bi-modal features corresponding to the first image are merged with the visual features of the first image, and a visual feature map in which the bi-modal features corresponding to the second image are merged with the visual features of the second image.
4 . The information processing apparatus according to claim 3 , wherein the encoder includes:
a first layer encoder configured to output the tri-modal features, the visual feature map related to the first image, and the visual feature map related to the second image; and
a second layer encoder configured to generate bi-modal features in which the visual feature maps that have been input from the first layer encoder and the language features of the text information are respectively fused, and configured to output tri-modal features in which the bi-modal features that have been generated are combined together.
5 . The information processing apparatus according to claim 4 , wherein a spatial size of the tri-modal features output from the second layer encoder is smaller than a spatial size of the tri-modal features output from the first layer encoder.
6 . The information processing apparatus according to claim 5 , wherein the decoder includes:
a second layer decoder configured to decode by using the tri-modal features output from the second layer encoder; and
a first layer decoder configured to decode features in which the tri-modal features output from the first layer encoder are incorporated into the features decoded by the second layer decoder.
7 . The information processing apparatus according to claim 1 , further comprising a classification model configured to classify a state of a subject displayed in the first image as a subtask, by using the tri-modal features.
8 . The information processing apparatus according to claim 2 , further comprising a classification model configured to classify a state of a subject displayed in the first image as a subtask, by using the bi-modal features corresponding to the second image.
9 . The information processing apparatus according to claim 4 , further comprising a classification model configured to classify a state of a subject displayed in the first image as a subtask, by using the tri-modal features generated by the second layer encoder.
10 . The information processing apparatus according to claim 4 , further comprising a classification model configured to classify a state of a subject displayed in the first image as a subtask, by using the bi-modal features generated by the second layer encoder.
11 . The information processing apparatus according to claim 7 , wherein the classification model configured to classify the state of the subject classifies at least any of whether the state of the subject is day or night, which time zone, what weather, and whether there is sunshine.
12 . The information processing apparatus according to claim 3 , wherein the encoder performs processing by using a language gate on the bi-modal features corresponding to the second image, adds the visual features of the second image, and outputs the visual feature map related to the second image.
13 . The information processing apparatus according to claim 1 , wherein the visual features of the first image are generated by inputting the first image into a transformer corresponding to the first image, and the visual features of the second image are generated by inputting the second image into a transformer corresponding to the second image.
14 . The information processing apparatus according to claim 1 , wherein the text information includes either utterance by voice or text that has been input.
15 . A learning method performed by an information processing apparatus for training at least one machine learning model to generate at least one trained machine learning model, wherein
the at least one machine learning model each includes a neural network, and the at least one machine learning model includes:
an encoder configured to receive inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, configured to generate bi-modal features corresponding to each image in which language features of the text information are fused with visual features of each image by use of an attention mechanism, and configured to output tri-modal features in which the bi-modal features that have been generated are combined together; and
a decoder configured to output a region on the first image corresponding to the designation of the place in the text information, by using either the tri-modal features or the bi-modal features corresponding to each image,
the learning method comprising
changing a weighting parameter of the neural network to reduce a value of a loss function using a loss with use of an output from the decoder and a correct answer indicating the region on the first image.
16 . The learning method according to claim 15 , wherein the at least one machine learning model further includes a classification model configured to classify a state of a subject displayed in the first image as a subtask, by using either features including the visual features of the second image or the tri-modal features,
the learning method further comprising
changing the weighting parameter of the neural network to reduce the value of the loss function using the loss with use of the output from the decoder and the correct answer indicating the region on the first image, and a loss with use of an output from the classification model and a correct answer indicating the state of the subject.
17 . A moving object comprising:
an image capturing apparatus;
an acquisition unit configured to acquire text information; and
at least one processor configured to perform processing of at least one trained machine learning model, wherein
the at least one trained machine learning model includes:
an encoder configured to receive inputs of text information including designation of a place, a first image that is an image captured by the image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and configured to output tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information; and
a decoder configured to output a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.
18 . An information processing method performed by an information processing apparatus for performing at least one trained machine learning model,
the information processing method comprising:
receiving inputs of text information including designation of a place, a first image that is an image captured by an image capturing apparatus and that includes the place, and a second image obtained by dividing a region for every identical object in the first image, and encoding tri-modal features that have been generated to include visual features of the first image that has been captured, visual features of the second image obtained by dividing the region, and language features of the text information; and
decoding a region on the first image corresponding to the designation of the place in the text information, by using the tri-modal features.