LEARNING DEVICE, LEARNING METHOD, AND LEARNING PROGRAM
A learning device includes processing circuitry configured to extract an encoding feature having a time series direction on a basis of input data of one or both of monomodal data that is data of a single modal or multimodal pair data including a plurality of different modals, embed segment information that is information for identifying a type of the modal of the input data in the encoding feature on a basis of a predetermined condition, connect, on a basis of input condition of a segment-embedded feature in which the segment information is embedded, a plurality of segment-embedded features in the time series direction as a modal-connected feature, and calculate a model parameter using an estimated vector of a cross-modal task estimated on a basis of one or both of the segment-embedded feature or the modal-connected feature and correct data.
1 . A learning device comprising:
processing circuitry configured to:
extract an encoding feature having a time series direction on a basis of input data of one or both of monomodal data that is data of a single modal or multimodal pair data including a plurality of different modals;
embed segment information that is information for identifying a type of the modal of the input data in the encoding feature on a basis of a predetermined condition;
connect, on a basis of input condition of a segment-embedded feature in which the segment information is embedded, a plurality of segment-embedded features in the time series direction as a modal-connected feature; and
calculate a model parameter using an estimated vector of a cross-modal task estimated on a basis of one or both of the segment-embedded feature or the modal-connected feature and correct data.
2 . The learning device according to claim 1 , wherein the processing circuitry is further configured to:
extract the single encoding feature in a case where the input data is the single monomodal data, and
extract the encoding feature according to a number of types of the modals included in the input data in a case where the input data is one or both of two or more of the monomodal data or the multimodal pair data.
3 . The learning device according to claim 2 , wherein the processing circuitry is further configured to
extract the encoding feature on a basis of a neural network corresponding to the type of the modal.
4 . The learning device according to claim 1 , wherein the processing circuitry is further configured to
embed, in the encoding feature, a vector having a same sequence length as the encoding feature as an input and including a fixed value different for each modal.
5 . The learning device according to claim 1 , wherein the processing circuitry is further configured to
in a case of having a plurality of the segment-embedded features as inputs, connect the plurality of the segment-embedded features in the time series direction.
6 . The learning device according to claim 1 , wherein the processing circuitry is further configured to
perform conversion using a function of an arbitrary neural network on a basis of one or both of the segment-embedded feature or the modal-connected feature, and estimate a vector corresponding to the correct data as the estimated vector of the cross-modal task.
7 . A learning method comprising:
extracting an encoding feature having a time series direction on a basis of input data of one or both of monomodal data that is data of a single modal or multimodal pair data including a plurality of different modals;
embedding segment information that is information for identifying a type of the modal of the input data in the encoding feature on a basis of a predetermined condition;
connecting, on a basis of input condition of a segment-embedded feature in which the segment information is embedded, a plurality of segment-embedded features in the time series direction as a modal-connected feature; and
calculating a model parameter using an estimated vector of a cross-modal task estimated on a basis of one or both of the segment-embedded feature or the modal-connected feature and correct data, by processing circuitry.
8 . A non-transitory computer-readable recording medium storing therein a learning program that causes a computer to execute a process comprising:
extracting an encoding feature having a time series direction on a basis of input data of one or both of monomodal data that is data of a single modal or multimodal pair data including a plurality of different modals;
embedding segment information that is information for identifying a type of the modal of the input data in the encoding feature on a basis of a predetermined condition;
connecting, on a basis of input condition of a segment-embedded feature in which the segment information is embedded, a plurality of segment-embedded features in the time series direction as a modal-connected feature; and
calculating a model parameter using an estimated vector of a cross-modal task estimated on a basis of one or both of the segment-embedded feature or the modal-connected feature and correct data.