System and method for detecting video semantic interval

1 . A system for detecting a video semantic interval, comprising:

a communication module configured to receive a video and a query sentence;

memory in which a program for outputting a semantic interval proposal from the video and the query sentence is stored; and

a processor configured to execute the program stored in the memory,

wherein by executing the program, the processor outputs a semantic interval proposal having start timing and end timing, which is matched with the query sentence within the video over a pre-trained video semantic interval detection network based on boundary refinements as results of a detection of the semantic interval proposal, and outputs a semantic interval proposal having a variable boundary through refinements of a predetermined semantic interval proposal, and

wherein the processor operates a proposal feature extraction unit comprising:

a unimodal feature encoding unit configured to extract unimodal features from the received video and query sentence, respectively;

a multimodal feature encoding unit configured to extract a multimodal feature in which the features of the received video and query sentence interact with each other;

a cross-modal proposal feature map extraction unit configured to extract a feature map of the semantic interval proposal based on the unimodal features and the multimodal feature;

a proposal interval length similarity map extraction unit configured to extract a proposal interval length similarity map based on the multimodal feature; and

a proposal interaction feature map extraction unit configured to concatenate the feature map of the semantic interval proposal and the proposal interval length similarity map and to extract a proposal interaction feature map through an interaction between a plurality of the semantic interval proposals.

2 . The system of claim 1 , wherein:

the unimodal feature encoding unit extracts a video feature by inputting the video to a pre-trained video feature extractor and extracts a sentence feature by inputting the query sentence to a pre-trained bidirectional encoder representations from transformer (BERT) model, and

the multimodal feature encoding unit generates a transformed video feature and a transformed sentence feature through a multi-head self-attention operation by adding positional embedding information to the video feature and concatenating the video feature and the sentence feature, and generates a guided sentence feature as the video feature by setting the video feature as a query, setting the sentence feature as a key and a value, and performing a multi-head cross-attention operation on the query and the key and value.

3 . The system of claim 2 , wherein the cross-modal proposal feature map extraction unit

generates feature information for each scale by performing a pooling process on the transformed video feature and the guided sentence feature, then extracting N segment clip features for each scale on the feature map of the semantic interval proposal, setting the N segment clip features as the key and value, setting a predetermined learnable query feature as the query, and performing a multi-head cross-attention operation on the key and value and the query,

generates information on a residual feature by performing a pooling process on a segment clip feature, then setting the segment clip feature as a key and a value, setting a predetermined learnable query feature as a query, and performing a multi-head cross-attention operation on the key and value and the query, and

generates a feature map for a specific scale by summing the feature information for each scale and the information on the residual feature and inputting results of summed results to a feed forward network (FFN), and generates the feature map of the semantic interval proposal for each of the transformed video feature and the guided sentence feature by generating a feature map for all scales.

4 . The system of claim 3 , wherein the cross-modal proposal feature map extraction unit

sets a proposal feature from the feature map of the semantic interval proposal as the query, sets the sentence feature as the key and value, performs the multi-head cross-attention operation on the query and the key and value, sums results of the multi-head cross-attention operation and the sentence feature on which average pooling has been performed, and inputs summed results to the FFN, and

generates a proposal guided sentence feature by summing an input and output of the FFN again and generates the cross-modal proposal feature map that has interacted with the sentence feature based on the proposal guided sentence feature and the proposal feature.

5 . The system of claim 2 , wherein the proposal interval length similarity map extraction unit

generates information (t s ) on a normalized length of a first query sentence with respect to the transformed sentence feature,

generates information (t v ) on a length of a second query sentence predicted based on a ground truth of the query sentence with respect to the transformed video feature, and

then calculates reliability based on the information on the lengths of the first and second query sentences.

6 . The system of claim 5 , wherein the proposal interval length similarity map extraction unit

generates information on a length of a normalized interval that is predicted for each proposal interval, and

extracts the proposal interval length similarity map by applying similarity that is calculated based on the information on the length of the normalized interval, the information on the length of the first query sentence, and the reliability to all proposal intervals.

7 . The system of claim 1 , wherein:

the proposal interaction feature map extraction unit comprises a plurality of convolution layers, and generates local interaction results between the feature map of the semantic interval proposal and the proposal interval length similarity map,

the proposal interaction feature map extraction unit comprises a convolution layer and a transformer and generates global interaction results between the feature map of the semantic interval proposal and the proposal interval length similarity map, and

the proposal interaction feature map extraction unit extracts the proposal interaction feature map by concatenating the local and global interaction results.

8 . The system of claim 2 , wherein the processor operates

a boundary matching unit configured to generate a candidate interval matching map based on the proposal interaction feature map,

a boundary refinement unit configured to generate boundary refinement maps indicative of a center offset and a length offset, respectively, for fine adjustment of a semantic interval proposal predicted based on the proposal interaction feature map, and

a proposal score map prediction unit configured to generate a final proposal score map based on the proposal interaction feature map, the candidate interval matching map, and the boundary refinement map.

9 . The system of claim 8 , wherein:

the processor trains the video semantic interval detection network by summing a plurality of loss functions, and

the plurality of loss functions comprises a semantic interval score loss function, a semantic interval length loss function, a proposal score loss function, and a proposal refinement loss function.

10 . The system of claim 9 , wherein the processor

labels a frame included in a ground truth interval of the query sentence with 1,

labels a frame not included in the ground truth interval with 0, and

calculates the semantic interval score loss function through a binary cross entropy loss operation between result values obtained by performing a linear operation and sigmoid operation on the frames labeled with 1 or 0, the transformed video feature, and the guided sentence feature.

11 . The system of claim 9 , wherein the processor calculates the semantic interval length loss function by performing a binary cross entropy loss operation on information on a normalized length of a first query sentence and information on a length of the ground truth interval of the query sentence with respect to the transformed sentence feature.

12 . The system of claim 9 , wherein the processor

generates normalized overlap information between the ground truth interval of the query sentence and each proposal interval of the video as label information,

calculates a loss function of the candidate interval matching map by applying a variable value of the label information based on a binary cross entropy loss operation,

calculates a loss function of the final proposal score map by applying a fixed value of the label information, and

then calculates the proposal score loss function by summing the loss functions of the candidate interval matching map and the final proposal score map.

13 . The system of claim 9 , wherein the processor

sets normalized overlap information between the ground truth interval of the query sentence and each proposal interval of the video as first label information,

sets a difference between central positions of the ground truth interval of the query sentence and each proposal interval of the video as second label information and calculates a loss function of the center offset by applying a fixed value of the first label information,

sets a difference between lengths of the ground truth interval of the query sentence and each proposal interval of the video as third label information and calculates a loss function of the length offset by applying the fixed value of the first label information,

calculates a loss function of overlap information between the ground truth interval of the query sentence and a proposal interval refined by the boundary refinement map by applying the fixed value of the first label information, and

then calculates the proposal refinement loss function by summing the loss functions of the center offset, the length offset, and the overlap information.

14 . The system of claim 8 , wherein the processor

searches for K semantic interval proposals each having a higher proposal score and not overlapping a preset threshold interval or more based on the final proposal score map, and

outputs a semantic interval proposal that is most matched with the query sentence, among the K semantic interval proposals.

15 . A method performed by a computer, comprising:

receiving a video and a query sentence;

inputting the video and the query sentence to a pre-trained video semantic interval detection network based on boundary refinements; and

outputting a semantic interval proposal having start timing and end timing, which is matched with the query sentence within the video, as results of a detection of the semantic interval proposal,

wherein the inputting comprises:

extracting unimodal features from the video and query sentence, respectively, using a unimodal feature encoding unit;

extracting a multimodal feature in which features of the video and query sentence interact with each other, using a multimodal feature encoding unit;

extracting a feature map of the semantic interval proposal based on the unimodal features and the multimodal feature, using a cross-modal proposal feature map extraction unit;

extracting a proposal interval length similarity map based on the multimodal feature, using a proposal interval length similarity map extraction unit; and

concatenating the feature map of the semantic interval proposal and the proposal interval length similarity map and extracting a proposal interaction feature map through an interaction between a plurality of the semantic interval proposals, using a proposal interaction feature map extraction unit; and

wherein the outputting of the semantic interval proposal as the results of the detection comprises outputting a semantic interval proposal having a variable boundary through refinements of a predetermined semantic interval proposal.