End-to-end multitask video retrieval with cross-attention

1 . A method comprising:

obtaining, at an electronic device, a video and a relational space-time query;

identifying, at the electronic device, at least one type of the relational space-time query, the at least one type of the relational space-time query representing an object type;

learning, at the electronic device, correlations among activities, objects, and time in the video using one or more cross-attention models;

obtaining, at the electronic device, one or more predictions generated using one or more outputs of the one or more cross-attention models based on the at least one type of the relational space-time query;

generating, at the electronic device, a response to the relational space-time query based on the one or more predictions;

splitting the video into multiple segments, wherein the one or more predictions are generated using the one or more outputs of the one or more cross-attention models for each of the multiple segments and splitting the video into the multiple segments comprises splitting the video into separate image frames; and

generating a list including the predictions for the multiple segments in order to generate an object prediction for the video.

2 . The method of claim 1 , wherein:

the at least one type of the relational space-time query further representing an activity type; and

the method further comprises combining the predictions for the multiple segments into a vector in order to generate an activity prediction for the video.

3 . The method of claim 1 , wherein:

the at least one type of the relational space-time further representing a time type;

each prediction is associated with a start time and an end time; and

the method further comprises stitching at least two of the predictions together based on at least some of their start and end times in order to generate a time prediction for the video.

4 . The method of claim 1 , wherein learning the correlations among the activities, objects, and time in the video comprises:

generating embeddings of at least some image frames in the video and embeddings of at least one of an activity or an object identified by the relational space-time query;

identifying one or more tokens for each of the at least some image frames in the video and for at least one of the activity or the object identified by the relational space-time query based on the embeddings;

performing at least one of activity cross-attention, object cross-attention, or time cross- attention based on the tokens; and

processing results of at least one of the activity cross-attention, object cross-attention, or time cross-attention in order to generate at least one of an activity prediction, an object prediction, or a time prediction.

5 . The method of claim 1 , wherein the one or more cross-attention models represent a single cross-attention model.

6 . An electronic device comprising:

at least one processing device configured to:

obtain a video and a relational space-time query;

identify at least one type of the relational space-time query, the at least one type of the relational space-time query representing an object type;

learn correlations among activities, objects, and time in the video using one or more cross-attention models;

obtain one or more predictions generated using one or more outputs of the one or more cross-attention models based on the at least one type of the relational space-time query;

generate a response to the relational space-time query based on the one or more predictions;

split the video into multiple segments, wherein the one or more predictions are generated using the one or more outputs of the one or more cross-attention models for each of the multiple segments and splitting the video into the multiple segments comprises splitting the video into separate image frames; and

generate a list including the predictions for the multiple segments in order to generate an object prediction for the video.

7 . The electronic device of claim 6 , wherein:

the at least one type of the relational space-time query further representing an activity type; and

the at least one processing device is further configured to combine the predictions for the multiple segments into a vector in order to generate an activity prediction for the video.

8 . The electronic device of claim 6 , wherein:

the at least one type of the relational space-time query further representing a time type;

each prediction is associated with a start time and an end time; and

the at least one processing device is further configured to stitch at least two of the predictions together based on at least some of their start and end times in order to generate a time prediction for the video.

9 . The electronic device of claim 6 , wherein, to learn the correlations among the activities, objects, and time in the video, the at least one processing device is configured to:

generate embeddings of at least some image frames in the video and embeddings of at least one of an activity or an object identified by the relational space-time query;

identify one or more tokens for each of the at least some image frames in the video and for at least one of the activity or the object identified by the relational space-time query based on the embeddings;

perform at least one of activity cross-attention, object cross-attention, or time cross- attention based on the tokens; and

process results of at least one of the activity cross-attention, object cross-attention, or time cross-attention in order to generate at least one of an activity prediction, an object prediction, or a time prediction.

10 . The electronic device of claim 6 , wherein the one or more cross-attention models represent a single cross-attention model.

11 . A non-transitory machine readable medium containing instructions that when executed cause at least one processor of an electronic device to:

obtain a video and a relational space-time query;

identify at least one type of the relational space-time query, the at least one type of the relational space-time query representing an object type;

learn correlations among activities, objects, and time in the video using one or more cross-attention models;

obtain one or more predictions generated using one or more outputs of the one or more cross-attention models based on the at least one type of the relational space-time query;

generate a response to the relational space-time query based on the one or more predictions;

split the video into multiple segments, wherein the one or more predictions are generated using the one or more outputs of the one or more cross-attention models for each of the multiple segments and splitting the video into the multiple segments comprises splitting the video into separate image frames; and

generate a list including the predictions for the multiple segments in order to generate an object prediction for the video.

12 . The non-transitory machine readable medium of claim 11 , wherein:

the at least one type of the relational space-time query further representing an activity type; and

the non-transitory machine readable medium further contains instructions that when executed cause the at least one processor to combine the predictions for the multiple segments into a vector in order to generate an activity prediction for the video.

13 . The non-transitory machine readable medium of claim 11 , wherein:

the at least one type of the relational space-time query further representing a time type;

each prediction is associated with a start time and an end time; and

the non-transitory machine readable medium further contains instructions that when executed cause the at least one processor to stitch at least two of the predictions together based on at least some of their start and end times in order to generate a time prediction for the video.

14 . The non-transitory machine readable medium of claim 11 , wherein the instructions that when executed cause the at least one processor to learn the correlations among the activities, objects, and time in the video comprise:

instructions that when executed cause the at least one processor to:

generate embeddings of at least some image frames in the video and embeddings of at least one of an activity or an object identified by the relational space-time query;

identify one or more tokens for each of the at least some image frames in the video and for at least one of the activity or the object identified by the relational space-time query based on the embeddings;

perform at least one of activity cross-attention, object cross-attention, or time cross-attention based on the tokens; and

process results of at least one of the activity cross-attention, object cross- attention, or time cross-attention in order to generate at least one of an activity prediction, an object prediction, or a time prediction.