Video processing method using transfer learning and pre-training server
There is provided a video processing method performed by a computing device, the method including the steps of: collecting video from an external device; generating preprocessed data by extracting two-dimensional or three-dimensional skeleton information from the video; pre-training a first artificial intelligence model including N transformer blocks from the preprocessed data by applying an attention from a body of an object to a plurality of joints, an attention from each of the plurality of joints to the body, and an attention between persons; and learning, when parameters determined as a result of the pre-training of the first artificial intelligence model are transferred, a method of recognizing an action from the video received from the external device, using a second artificial intelligence model including the N transformer blocks on the basis of the parameters, wherein N is a natural number equal to or larger than 2.
1 . A video processing method performed by a computing device, the method comprising the steps of:
collecting video from an external device;
generating preprocessed data by extracting two-dimensional or three-dimensional skeleton information from the video;
pre-training a first artificial intelligence model including N transformer blocks from the preprocessed data by applying an attention from a body of an object to a plurality of joints, an attention from each of the plurality of joints to the body, and an attention between persons; and
learning, when parameters determined as a result of the pre-training of the first artificial intelligence model are transferred, a method of recognizing an action from the video received from the external device, using a second artificial intelligence model including the N transformer blocks on the basis of the parameters, wherein
N is a natural number equal to or larger than 2.
2 . The method according to claim 1 , wherein the step of pre-training a first artificial intelligence model includes the steps of:
applying a slice of a positional embedding tensor corresponding to each frame of an output of a previous block, and performing first layer normalization, by an n-th transformer block among the N transformer blocks;
applying a spatial multi-head attention (MHA) to a result of the first layer normalization, by the n-th transformer block; and
deriving a first result by adding a result of applying the slice of a positional embedding tensor corresponding to each frame of an output of a previous block to a result of applying the spatial MHA, by the n-th transformer block, wherein
n is a natural number between 2 and N.
3 . The method according to claim 2 , further comprising the steps of:
applying a matrix with a changed dimension of the positional embedding tensor to a pose sequence matrix in which the first result corresponding to each frame is stacked, and performing second layer normalization;
applying a temporal MHA to a result of the second layer normalization; and
deriving a second result by adding a result of applying the matrix with a changed dimension of the positional embedding tensor to a pose sequence matrix to a result of applying the temporal MHA.
4 . The method according to claim 3 , further comprising the steps of:
performing third layer normalization on the second result;
applying a multi-layer perceptron (MLP) to a result of the third layer normalized; and
deriving a third result by adding the second result to a result of applying the MLP.
5 . The method according to claim 1 , further comprising the step of deriving a motion sequence representation for an input motion sequence according to the video, by an N-th block among the N transformer blocks.
6 . A pre-training server comprising:
a processor;
a memory; and
a computer program loaded on the memory and executed by the processor, wherein
the computer program includes:
an instruction for collecting video from an external device;
an instruction for generating preprocessed data by extracting two-dimensional or three-dimensional skeleton information from the video;
an instruction for pre-training a first artificial intelligence model including N transformer blocks from the preprocessed data by applying an attention from a body of an object to a plurality of joints, an attention from each of the plurality of joints to the body, and an attention between persons; and
an instruction for learning, when parameters determined as a result of the pre-training of the first artificial intelligence model are transferred, a method of recognizing an action from the video received from the external device, using a second artificial intelligence model including the N transformer blocks on the basis of the parameters, wherein
N is a natural number equal to or larger than 2.