Apparatus and method for extracting qualitative characteristics of video
View Patent ↗A method, a device and a recording medium for extracting a qualitative characteristic of a video may include segmenting a video into at least one video segment, and based on an edge representing a relationship between objects of the video segment and the objects, generating video graph sequences in a graph form of the video segment, and generation of the video graph sequences may be performed based on an object feature vector expressing a qualitative characteristic of the objects as a vector and an edge feature vector expressing a qualitative characteristic of the edge as a vector.
1 . A method for extracting a qualitative characteristic of a video, the method comprising:
dividing a video into at least one video segment;
generating, for each video segment, based on objects included in the video segment and edges representing relationships among the objects, a video graph sequence in a graph form of the video segment, wherein generating the video graph sequence includes generating an object feature vector expressing qualitative characteristics of the objects as a vector and an edge feature vector expressing a qualitative characteristic of each edge as a vector;
embedding each generated video graph sequence into a structured graph representation suitable for processing by a graph neural network;
classifying, using the graph neural network, each embedded representation of the video graph sequence into a qualitative characteristic;
grouping a plurality of video graph sequences having similar qualitative characteristics into a single style; and
based on the style, generating a feature matrix by extracting the object feature vector and the edge feature vector having a value equal to or greater than a specific threshold value from video graph sequences belonging to a same style.
2 . The method of claim 1 , wherein the video graph sequence is generated by a matrix operation of an object characteristic matrix and an object adjacency matrix.
3 . The method of claim 2 , wherein the object characteristic matrix is a k×n matrix generated based on the object feature vector,
the k is a number of the objects,
the n is a number of features of the object feature vector.
4 . The method of claim 3 , wherein the object adjacency matrix is a k×k matrix generated based on the edge feature vector.
5 . The method of claim 1 , further comprising:
identifying a style of the video segment based on the classified qualitative characteristic; and
storing the video segment in a database based on the identified style of the video segment.
6 . The method of claim 1 , wherein the qualitative characteristic of the objects includes a color, a texture, an age and a gender of the objects.
7 . The method of claim 1 , wherein the qualitative characteristic of the edge includes a geographical relationship and a behavioral relationship between the objects.
8 . The method of claim 1 , wherein the objects include an object, a person and a background geographical feature which form the video segment.
9 . A non-transitory computer-readable recording medium, wherein a computer program for executing a method according to claim 1 in a computer is recorded.
10 . A device for extracting a qualitative characteristic of a video, the device comprising:
a processor; and a memory having instructions stored thereon, which, when executed by the processor, cause the processor to perform:
a video segmentation module which divides a video into at least one video segment; and
a graph embedding module which generates, for each video segment, a video graph sequence in a graph form of the video segment based on objects included in the video segment and edges representing relationships among the objects, and embeds each generated video graph sequence into a structured graph representation suitable for processing by a graph neural network, wherein the video graph sequence is generated based on an object feature vector expressing qualitative characteristics of the objects as a vector and an edge feature vector expressing a qualitative characteristic of each edge as a vector;
a video classification module which classifies, using the graph neural network, each embedded representation of the video graph sequence into a qualitative characteristic, groups a plurality of video graph sequences having similar qualitative characteristics to into a single style, and based on the style, generates a feature matrix by extracting the object feature vector and the edge feature vector having a value equal to or greater than a specific threshold value from video graph sequences belonging to a same style.
11 . The device of claim 10 , wherein the video graph sequence is generated by a matrix operation of an object characteristic matrix and an object adjacency matrix.
12 . The device of claim 11 , wherein the object characteristic matrix is a k×n matrix generated based on the object feature vector,
the k is a number of the objects,
the n is a number of features of the object feature vector.
13 . The device of claim 12 , wherein the object adjacency matrix is a k×k matrix generated based on the edge feature vector.
14 . The device of claim 10 , wherein the video classification module includes a video style tagging unit which identifies a style of the video segment based on the classified qualitative characteristic and stores the video segment in a database based on the identified style of the video segment.
15 . The device of claim 10 , wherein the qualitative characteristic of the objects includes a color, a texture, an age and a gender of the objects.
16 . The device of claim 10 , wherein the qualitative characteristic of the edge includes a geographical relationship and a behavioral relationship between the objects.