Information processing device, information processing method, and information processing program

1 . An information processing device, comprising:

one or more processors; and memory storing one or more programs, the one or more programs including instructions that, when executed by the one or more processors, cause the one or more processors to:

acquire a video image;

partition the acquired video image into a plurality of event video images as candidates for appending a caption by partitioning according to events;

select an event video image from the partitioned event video image candidates by using a selection model such that a range of an event is neither too broad nor too narrow, wherein the selection model selects the event video image using a differentiable function;

generate a video image set with captions by employing an appending model for appending a caption to an event represented by the input event video image to append a caption to the selected event video image;

store a feature of an event video image selected in the past from the candidates in a first memory vector, and when selecting the event video image, employ the first memory vector to select an event video image indicating a continuation of an event video image selected in the past;

store a feature value of the caption appended to the event video image in a second memory vector;

employ the second memory vector to select a word indicating a continuation of an appended caption;

add the selected word to a caption appended in the past and append this to the event video image; and

update the first memory vector using the second memory vector, and use the updated first memory vector to select a next event video image from the candidates.

2 . The information processing device of claim 1 , wherein the differentiable function includes a Gumbel-Softmax function.

3 . The information processing device of claim 2 , further comprising:

training by propagating a training result leaned by the appending model to the selection model, or training each of the selection model and the appending model individually.

4 . The information processing device of claim 2 , the one or more processors are further configured to:

update the second memory vector, using the first memory vector, and

select a next word for the selected event video image using the updated second memory vector.

5 . The information processing device of claim 2 , the one or more processors are further configured to:

training by propagating a training result learned by the appending model to the selection model, or training each of the selection model and the appending model individually.

6 . The information processing device of claim 5 , wherein:

partitioning the acquired video image is performed by a partitioning model that has been trained to partition the event video image from the video image; and

training by propagating the training result learned by the selection model to the partitioning model.

7 . The information processing device of claim 1 , wherein the one or more processors are further configured to:

update the second memory vector, using the first memory vector, and

select a next word for the selected event video image using the updated second memory vector.

8 . The information processing device of claim 7 , further comprising:

training by propagating a training result learned by the appending model to the selection model, or training each of the selection model and the appending model individually.

9 . The information processing device of claim 1 , wherein the one or more processors are further configured to:

train by propagating a training result learned by the appending model to the selection model, or train each of the selection model and the appending model individually.

10 . The information processing device of claim 9 , wherein:

partitioning the acquired video image is performed by a partitioning model that has been trained to partition the event video image from the video image; and

training by propagating the training result learned by the selection model to the partitioning model.

11 . An information processing method comprising:

at a computer system that includes one or more processors and memory:

acquiring a video image;

partitioning the acquired video image into a plurality of event video images as candidates for appending a caption by partitioning according to events;

selecting an event video image from the partitioned event video image candidates by using a selection model such that a range of an event is neither too broad nor too narrow, wherein the selection model selects the event video image using a differentiable function;

generating a video image set with captions by employing an appending model for appending a caption to an event represented by the input event video image to append a caption to the selected event video image;

storing a feature of an event video image selected in the past from the candidates in a first memory vector, and when selecting the event video image, employing the first memory vector to select an event video image indicating a continuation of an event video image selected in the past; storing a feature value of the caption appended to the event video image in a second memory vector;

employing the second memory vector to select a word indicating a continuation of an appended caption;

adding the selected word to a caption appended in the past and appending this to the event video image; and

updating the first memory vector stored in the storage unit, using the second memory vector, to select a next event video image from the candidates.

12 . The information processing method of claim 11 , wherein the differentiable function includes a Gumbel-Softmax function.

13 . The information processing method of claim 11 , further comprising:

updating the second memory vector stored in the storage unit, using the first memory vector, and

selecting a next word for the selected event video image using the updated second memory vector.

14 . The information processing method claim 11 , further comprising:

training by propagating a training result learned by the appending model to the selection model, or individually training each of the selection model and the appending model.

15 . The information processing method of claim 14 , wherein:

partitioning the acquired video image is performed by a partitioning model that has been trained to partition the event video image from the video image; and

training by propagating the training result learned by the selection model to the partitioning model.

16 . A non-transitory computer-readable storage medium storing an information processing program that causes processing to be executed by a computer, the processing comprising:

acquiring a video image;

partitioning the acquired video image into a plurality of event video images as candidates for appending a caption by partitioning according to events;

selecting an event video image from the partitioned event video image candidates by using a selection model such that a range of an event is neither too broad nor too narrow, wherein the selection model selects the event video image using a differentiable function;

generating a video image set with captions by employing an appending model for appending a caption to an event represented by the input event video image to append a caption to the selected event video image;

storing a feature of an event video image selected in the past from the candidates in a first memory vector, and when selecting the event video image, employing the first memory vector to select an event video image indicating a continuation of an event video image selected in the past;

storing a feature value of the caption appended to the event video image in a second memory vector;

employing the second memory vector to select a word indicating a continuation of an appended caption;

adding the selected word to a caption appended in the past and appending this to the event video image; and

updating the first memory vector stored in the storage unit, using the second memory vector, to select a next event video image from the candidates.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the differentiable function includes a Gumbel-Softmax function.

18 . The non-transitory computer-readable storage medium of claim 16 , further comprising:

updating the second memory vector stored in the storage unit, using the first memory vector, and

selecting a next word for the selected event video image using the updated second memory vector.

19 . The non-transitory computer-readable storage medium of claim 16 , further comprising:

training by propagating a training result learned by the appending model to the selection model, or individually training each of the selection model and the appending model.

20 . The non-transitory computer-readable storage medium of claim 19 , wherein:

partitioning the acquired video image is performed by a partitioning model that has been trained to partition the event video image from the video image; and

training by propagating the training result learnt by the selection model to the partitioning model.