Video processing device and method
A video processing apparatus includes a memory storing instructions, and at least one processor configured to execute the instructions to generate a plurality of feature information by analyzing a video signal comprising a plurality of images based on a first DNN, extract a first altitude component and a first planar component corresponding to a movement of an object in a video from the video signal based on a second DNN, extract a second planar component corresponding to a movement of a sound source in audio from a first audio signal based on a third DNN, generate a second altitude component based on the first altitude component, the first planar component, and the second planar component, output a second audio signal comprising the second altitude component based on the feature information, and synchronize the second audio signal with the video signal and output the synchronized second audio signal and video signal.
1. A video processing apparatus comprising:
a memory storing at least one instruction; and
at least one processor configured to execute the at least one instruction to:
generate a plurality of feature information for time and frequency by analyzing a video signal comprising a plurality of images, based on a first deep neural network (DNN);
extract a first altitude component corresponding to a movement of an object in a vertical direction in a video and a first planar component corresponding to the movement of the object in a horizontal direction in the video from the video signal, based on a second DNN;
extract a second planar component corresponding to a movement of a sound source in the horizontal direction in audio from a first audio signal, based on a third DNN;
generate a second altitude component corresponding to the movement of the sound source in the vertical direction in the audio based on the first altitude component, the first planar component, and the second planar component;
output a second audio signal comprising the second altitude component, based on the plurality of feature information; and
synchronize the second audio signal with the video signal and output the synchronized second audio signal and video signal.
2. The video processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
synchronize the video signal with the first audio signal;
generate M pieces of one-dimensional image feature map information corresponding to the movement of the object in the video from the video signal by using the first DNN, M being an integer greater than or equal to 1; and
generate the plurality of feature information for time and frequency by performing tiling related to frequency on the M pieces of one-dimensional image feature map information, the plurality of feature information including the M pieces of one-dimensional image feature map information for time and frequency.
3. The video processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
synchronize the video signal with the first audio signal:
extract N+M pieces of feature map information corresponding to the movement of the object in the horizontal direction in the video with respect to time from the video signal by using a (2-1st) DNN, N and M being integers greater than or equal to 1;
extract N+M pieces of feature map information corresponding to the movement of the object in the vertical direction in the video with respect to time from the video signal by using a (2-2nd) DNN, wherein the (2-1st) DNN and the (2-2nd) DNN are included in the second DNN and are different from each other;
extract N+M pieces of feature map information corresponding to the movement of the sound source in the horizontal direction in the audio from the first audio signal by using the third DNN;
generate N+M pieces of correction map information with respect to time corresponding to the second altitude component based on the N+M pieces of feature map information corresponding to the movement of the object in the horizontal direction in the video, the N+M pieces of feature map information corresponding to the movement of the object in the vertical direction in the video, and the N+M pieces of feature map information corresponding to the movement of the sound source in the horizontal direction in the audio; and
generate N+M pieces of correction map information with respect to time and frequency corresponding to the second altitude component by performing tiling related to frequency on the N+M pieces of correction map information with respect to time.
4. The video processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
generate time and frequency information for a 2-channel by performing frequency conversion operation on the first audio signal;
generate N pieces of audio feature map information with respect to time and frequency from the time and frequency information for the 2-channel by using a (4-1st) DNN, N being an integer greater than or equal to 1;
generate N+M pieces of audio and image integrated feature map information based on M pieces of image feature map information with respect to time and frequency included in the plurality of feature information for time and frequency and the N pieces of audio feature map information with respect to time and frequency;
generate a frequency domain second audio signal for n-channel (where, n is an integer greater than 2) from the N+M pieces of audio and image integrated feature map information by using a (4-2nd) DNN;
generate an audio correction map information for the n-channel from N+M pieces of correction map information with respect to time and frequency corresponding to the N+M pieces of audio/image integrated feature map information and the second altitude component by using a (4-3rd) DNN;
generate a corrected frequency domain second audio signal for the n-channel by performing correction on the frequency domain second audio signal for the n-channel based on the audio correction map information for the n-channel; and
output the second audio signal for the n-channel by inversely frequency converting the corrected frequency domain second audio signal for the n-channel, and
wherein the (4-1st) DNN, the (4-2nd) DNN and the (4-3rd) DNN are included in a fourth DNN for outputting the second audio signal and are different from each other.
5. The video processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to output the second audio signal based on a fourth DNN for outputting the second audio signal,
wherein the first DNN is a DNN for generating the plurality of feature information for time and frequency, the second DNN is a DNN for extracting the first altitude component and the first planar component, the third DNN is a DNN for extracting the second planar component, and
wherein the at least one processor is further configured to execute the at least one instruction to train the first DNN, the second DNN, the third DNN and the fourth DNN according to a result of comparison of a first frequency domain training reconstruction three-dimensional audio signal reconstructed based on a first training two-dimensional audio signal and a first training image signal with a first frequency domain training three-dimensional audio signal obtained by frequency converting a first training three-dimensional audio signal.
6. The video processing apparatus of claim 5 , wherein the at least one processor is further configured to execute the at least one instruction to:
determine generation loss information by comparing the first frequency domain training reconstruction three-dimensional audio signal with the first frequency domain training three-dimensional audio signal, and
update parameters of the first DNN, the second DNN, the third DNN and the fourth DNN based on the generation loss information.
7. The video processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to output the second audio signal based on a fourth DNN for outputting the second audio signal,
wherein the first DNN is a DNN for generating the plurality of feature information for time and frequency, the second DNN is a DNN for extracting the first altitude component and the first planar component, and the third DNN is a DNN for extracting the second planar component, and
wherein the at least one processor is further configured to execute the at least one instruction to train the first DNN, the second DNN, the third DNN, and the fourth DNN according to a result of comparison of a frequency domain training reconstruction three-dimensional audio signal reconstructed based on a first training two-dimensional audio signal, a first training image signal and a user input parameter information with a first frequency domain training three-dimensional audio signal obtained by frequency converting a first training three-dimensional audio signal.
8. The video processing apparatus of claim 7 , wherein the at least one processor is further configured to execute the at least one instruction to:
determine generation loss information by comparing the frequency domain training reconstruction three-dimensional audio signal with the first frequency domain training three-dimensional audio signal, and
update parameters of the first DNN, the second DNN, the third DNN, and the fourth DNN based on the generation loss information.
9. The video processing apparatus of claim 5 , wherein the first training two-dimensional audio signal and the first training image signal are obtained from a portable terminal, and
wherein the first training three-dimensional audio signal is obtained from an ambisonic microphone of the portable terminal.
10. The video processing apparatus of claim 5 , wherein parameter information of the first DNN, the second DNN, the third DNN, and the fourth DNN obtained as a result of training of the first DNN, the second DNN, the third DNN, and the fourth DNN is stored in the video processing apparatus or is received from a terminal connected to the video processing apparatus.
11. A video processing method of a video processing apparatus, the video processing method comprising:
generating a plurality of feature information for time and frequency by analyzing a video signal comprising a plurality of images based on a first deep neural network (DNN);
extracting a first altitude component corresponding to a movement of an object in a vertical direction in a video and a first planar component corresponding to the movement of the object in a horizontal direction in the video, from the video signal based on a second DNN;
extracting a second planar component corresponding a movement of a sound source in the horizontal direction in an audio from a first audio signal, based on a third DNN;
generating a second altitude component corresponding to the movement of the sound source in the vertical direction in the audio based on the first altitude component, the first planar component, and the second planar component;
outputting a second audio signal comprising the second altitude component, based on the plurality of feature information; and
synchronizing the second audio signal with the video signal and outputting the synchronized second audio signal and video signal.
12. The video processing method of claim 11 , wherein the generating the plurality of feature information for time and frequency comprises:
synchronizing the video signal with the first audio signal;
generating M pieces of one-dimensional image feature map information corresponding to the movement of the object in the video from the video signal by using the first DNN, M being an integer greater than or equal to 1; and
generating the plurality of feature information for time and frequency by performing tiling related to frequency on the M pieces of one-dimensional image feature map information, the plurality of feature information including M pieces of image feature map information for time and frequency.
13. The video processing method of claim 11 , wherein the extracting the first altitude component and the first planar component based on the second DNN and the extracting of the second planar component based on the third DNN comprise:
synchronizing the video signal with the first audio signal;
extracting N+M pieces of feature map information corresponding to the movement of the object in the horizontal direction in the video with respect to time from the video signal by using a (2-1st) DNN, N and M being integers greater than or equal to 1;
extracting N+M pieces of feature map information corresponding to the movement of the object in the vertical direction in the video with respect to time from the video signal by using a (2-2nd) DNN, wherein the (2-1st) DNN and the (2-2nd) DNN are included in the second DNN and are different from each other;
extracting N+M pieces of feature map information corresponding to the movement of the sound source in the horizontal direction in the audio from the first audio signal by using the third DNN, and
wherein the generating of the second altitude component based on the first altitude component, the first planar component, and the second planar component comprises:
generating N+M pieces of correction map information with respect to time corresponding to the second altitude component based on the N+M pieces of feature map information corresponding to the movement of the object in the horizontal direction in the video, the N+M pieces of feature map information corresponding to the movement of the object in the vertical direction, and the N+M pieces of feature map information corresponding to the movement of the sound source in the horizontal direction in the audio; and
generating N+M pieces of correction map information with respect to time and frequency corresponding to the second altitude component by performing tiling related to a frequency on the N+M pieces of correction map information with respect to time.
14. The video processing method of claim 11 , wherein the outputting the second audio signal comprising the second altitude component based on the plurality of feature information comprises:
obtaining time and frequency information for a 2-channel by performing frequency conversion operation on the first audio signal;
generating, from the time and frequency information for the 2-channel, N pieces of audio feature map information with respect to time and frequency by using a (4-1st) DNN, N being an integer greater than or equal to 1;
generating N+M pieces of audio and image integrated feature map information based on M pieces of image feature map information with respect to time and frequency included in the plurality of feature information for time and frequency and the N pieces of audio feature map information with respect to time and frequency;
generating a frequency domain second audio signal for n-channel (wherein, n is an integer greater than 2) from the N+M pieces of audio and image integrated feature map information by using a (4-2nd) DNN;
generating a audio correction map information with respect to the n-channel corresponding to the second altitude component from the N+M pieces of audio/image integrated feature map information by using a (4-3rd) DNN;
generating a corrected frequency domain second audio signal for the n-channel by performing correction on the frequency domain second audio signal for the n-channel based on the audio correction map information for the n-channel; and
outputting the second audio signal for the n-channel by inversely frequency converting the corrected frequency domain second audio signal, and
wherein the (4-1st) DNN, the (4-2nd) DNN and the (4-3rd) DNN are included in a fourth DNN for outputting the second audio signal and are different from each other.
15. The video processing method of claim 11 , wherein the outputting of the second audio signal comprises outputting the second audio signal based on a fourth DNN for outputting the second audio signal,
wherein the first DNN is a DNN for generating the plurality of feature information for each time and frequency, the second DNN is a DNN for extracting the first altitude component and the first planar component, and the third DNN is a DNN for extracting the second planar component, and
wherein the video processing method further comprises training the first DNN, the second DNN, the third DNN, and the fourth DNN according to a result of comparison of a first frequency domain training reconstruction three-dimensional audio signal reconstructed based on a first training two-dimensional audio signal and a first training image signal with a first frequency domain training three-dimensional audio signal obtained by frequency converting a first training three-dimensional audio signal.
16. The video processing method of claim 15 , further comprising:
determining generation loss information by comparing the first frequency domain training reconstruction three-dimensional audio signal with the first frequency domain training three-dimensional audio signal, and
updating parameters of the first DNN, the second DNN, the third DNN, and the fourth DNN based on the generation loss information.
17. The video processing apparatus of claim 15 , wherein parameter information of the first DNN, the second DNN, the third DNN, and the fourth DNN obtained as a result of training of the first DNN, the second DNN, the third DNN, and the fourth DNN is stored in the video processing apparatus or is received from a terminal connected to the video processing apparatus.
18. The video processing method of claim 11 , wherein the outputting the second audio signal comprises outputting the second audio signal based on a fourth DNN for outputting the second audio signal,
wherein the first DNN is a DNN for generating the plurality of feature information for each time and frequency, the second DNN is a DNN for extracting the first altitude component and the first planar component, and the third DNN is a DNN for extracting the second planar component, and
wherein the method further comprises training the first DNN, the second DNN, the third DNN, and the fourth DNN according to a result of comparison of a first frequency domain training reconstruction three-dimensional audio signal reconstructed based on a first training two-dimensional audio signal, a first training image signal and user input information with a first frequency domain training three-dimensional audio signal obtained by frequency converting a first training three-dimensional audio signal.
19. The video processing method of claim 18 , further comprising:
determining generation loss information by comparing the first frequency domain training reconstruction three-dimensional audio signal with the first frequency domain training three-dimensional audio signal, and
updating parameters of the first to fourth DNNs based on the generation loss information.
20. A non-transitory computer-readable recording medium having recorded thereon a program that is executable by a processor to perform the method of claim 11 .