Method for providing video and electronic device supporting the same
An electronic device is provided. The electronic device includes a memory, and at least one processor electrically connected to the memory, wherein the at least one processor is configured to obtain a video including an image and an audio, obtain information on at least one object included in the image from the image, obtain a visual feature of the at least one object, based on the image and the information on the at least one object, obtain a spectrogram of the audio, obtain an audio feature of the at least one object from the spectrogram of the audio, combine the visual feature and the audio feature, obtain, based on the combined visual feature and audio feature, information on a position of the at least one object the information indicating the position of the at least one object in the image, obtain an audio part corresponding to the at least one object in the audio, based on the combined visual feature and audio feature, and store, in the memory, the information on the position of the at least one object and the audio part corresponding to the at least one object.
1. An electronic device comprising:
at least one processor including processing circuitry; and
memory storing instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain a video including an image and an audio,
obtain information on at least one object included in the image from the image, the information on the at least one object including a map in which the at least one object included in the image is masked,
obtain a visual feature of the at least one object, based on the image and the information on the at least one object,
obtain a spectrogram of the audio,
obtain an audio feature of the at least one object from the spectrogram of the audio,
combine the visual feature and the audio feature,
obtain, based on the combined visual feature and audio feature, information on a position of the at least one object, the information indicating the position of the at least one object in the image, by obtaining a mask having a value of possibility that each pixel represents the at least one object, based on the combined visual feature and audio feature,
obtain an audio part corresponding to the at least one object in the audio, based on the combined visual feature and audio feature,
store, in the memory, the information on the position of the at least one object and the audio part corresponding to the at least one object,
display the video, and
while displaying the video, displaying an audio volume indicator and a time interval indicator for the audio part, the time interval indicator having a first portion displayed having a first visual attribute and a remaining portion of the time interval indicator displayed having a second visual attribute, the first visual attribute being a same as a visual attribute of the at least one object while the audio part corresponding to the at least one object is output, wherein the first visual attribute is different from the second visual attribute.
2. The electronic device of claim 1 , wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to combine the visual feature and the audio feature by performing an add operation, a multiplication operation, or a concatenation operation for the visual feature and the audio feature.
3. The electronic device of claim 1 , wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain an image having a value of possibility that each pixel represents an audio corresponding to the at least one object, based on the combined visual feature and audio feature, and
obtain an audio part corresponding to the at least one object in the audio, based on the obtained image.
4. The electronic device of claim 3 , wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to obtain an audio part corresponding to the at least one object in the audio, based on performing an AND operation for the spectrogram of the audio and the obtained image.
5. The electronic device of claim 1 ,
wherein the instructions that, when executed by the at least one processor individually or collectively, further cause the electronic device to perform training to generate an artificial intelligence model, and
wherein, to perform the training, the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to:
obtain multiple videos,
obtain information on at least one object from each of images of the multiple videos,
obtain a visual feature of each of the at least one object,
obtain a spectrogram of an audio corresponding to the at least one object,
obtain an audio feature of each of the at least one object,
combine the audio feature and the visual feature for each of the at least one object,
obtain, based on the combined visual feature and audio feature, information on a position of the at least one object, the information indicating the position of the at least one object in each of the images, and
obtain an audio part corresponding to the at least one object in the audio, based on the combined visual feature and audio feature.
6. The electronic device of claim 5 , wherein the instructions that, when executed by the at least one processor individually or collectively, cause the electronic device to generate an artificial intelligence model related to a segmentation artificial intelligence network, based on images of the multiple videos, and an image part of the at least one object in each of the images as a ground truth.
7. The electronic device of claim 5 , wherein, to obtain the information on the position of the at least one object, the instructions that, when executed by the at least one processor individually or collectively, further cause the electronic device to correct an error such that a distance between the audio feature and the visual feature is minimized by using a loss function based on metric learning.
8. The electronic device of claim 5 , wherein, to obtain the audio part corresponding to the at least one object in the audio, the instructions that, when executed by the at least one processor individually or collectively, further cause the electronic device to correct an error such that by using a loss function and for each of the at least one object, a pixel-specific distance between a feature obtained by combining the visual feature and the audio feature, and the spectrogram of the audio corresponding to the at least one object is minimized.
9. A method for providing a video by an electronic device, the method comprising:
obtaining a video including an image and an audio;
obtaining information on at least one object included in the image from the image, the information on the at least one object including a map in which the at least one object included in the image is masked;
obtaining a visual feature of the at least one object, based on the image and the information on the at least one object;
obtaining a spectrogram of the audio;
obtaining an audio feature of the at least one object from the spectrogram of the audio;
combining the visual feature and the audio feature;
obtaining, based on the combined visual feature and audio feature, information on a position of the at least one object, the information indicating the position of the at least one object in the image, by obtaining a mask having a value of possibility that each pixel represents the at least one object, based on the combined visual feature and audio feature;
obtaining an audio part corresponding to the at least one object in the audio, based on the combined visual feature and audio feature;
storing, in a memory of the electronic device, the information on the position of the at least one object and the audio part corresponding to the at least one object,
display the video; and
while displaying the video, displaying an audio volume indicator and a time interval indicator for the audio part, the time interval indicator having a first portion displayed having a first visual attribute and a remaining portion of the time interval indicator is displayed having a second visual attribute, the first visual attribute being a same as a visual attribute of the at least one object while the audio part corresponding to the at least one object is output, wherein the first visual attribute is different from the second visual attribute.
10. The method of claim 9 , wherein the combining of the visual feature and the audio feature comprises combining the visual feature and the audio feature by performing an add operation, a multiplication operation, or a concatenation operation for the visual feature and the audio feature.
11. The method of claim 10 , wherein the obtaining of the audio part corresponding to the at least one object comprises:
obtaining an image having a value of possibility that each pixel represents an audio corresponding to the at least one object, based on the combined visual feature and audio feature; and
obtaining an audio part corresponding to the at least one object in the audio, based on the obtained image.
12. The method of claim 11 , wherein the obtaining of the audio part corresponding to the at least one object in the audio comprises obtaining an audio part corresponding to the at least one object in the audio, based on performing an AND operation for the spectrogram of the audio and the obtained image.
13. The method of claim 9 , further comprising:
performing training to generate an artificial intelligence model,
wherein the performing of the training comprises:
obtaining multiple videos;
obtaining information on at least one object from each of images of the multiple videos;
obtaining a visual feature of each of the at least one object;
obtaining a spectrogram of an audio corresponding to the at least one object;
obtaining an audio feature of each of the at least one object;
combining the audio feature and the visual feature for each of the at least one object;
obtaining, based on the combined visual feature and audio feature, information on a position of the at least one object, the information indicating the position of the at least one object in each of the images; and
obtaining an audio part corresponding to the at least one object in the audio, based on the combined visual feature and audio feature.
14. The method of claim 13 , wherein the obtaining of the information on the at least one object comprises generating an artificial intelligence model related to a segmentation artificial intelligence network, based on images of the multiple videos, and an image part of the at least one object in each of the images as a ground truth.
15. The method of claim 13 , wherein the obtaining of the information on the position of the at least one object further comprises correcting an error such that a distance between the audio feature and the visual feature is minimized by using a loss function based on metric learning.
16. The method of claim 9 , wherein the obtaining of the audio part corresponding to the at least one object in the audio further comprises correcting an error such that by using a loss function and for each of the at least one object, a pixel-specific distance between a feature obtained by combining the visual feature and the audio feature, and the spectrogram of the audio corresponding to the at least one object is minimized.
17. The method of claim 9 , further comprising:
while displaying the video, displaying a second object, the time interval indicator having a third portion displayed having a third visual attribute, the second object being displayed having the third visual attribute.
18. The method of claim 9 , wherein the first visual attribute comprises at least one of an effect, a color, or a highlight.