Granted Patent B2

US 12,347,459 · App. 17/768,462 · Granted Jul 1, 2025

Inventors: Wei Zheng (Beijing, CN); Weiwei Lyu (Beijing, CN)

Assignee: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.

G11B27/031G06F3/0487

Abstract

Claims (78)

1. A method for generating a video file, comprising:

presenting, in response to a received video editing instruction, a video editing interface, wherein the video editing interface is configured to enable a selection of an image and a selection of an audio;

determining a target audio and a target image in response to receiving input via the video editing interface;

obtaining, for each of audio frames of the target audio, an audio parameter corresponding to the audio frame;

generating, for each of the audio frames, a spectrogram corresponding to the audio frame based on the obtained audio parameter;

generating, based on the generated spectrogram and the target image, a plurality of video frame images, wherein each of the plurality of video frame images corresponds to one of the audio frames, and each of the plurality of video frame images comprises an image of the spectrum corresponding to the one of the audio frames; and

performing, based on the plurality of video frame images and the target audio, video encoding to generate a target video file.

2. The method according to claim 1 , wherein the generating, for each of the audio frames, a spectrogram corresponding to the audio frame based on the obtained audio parameter comprises:

sampling the target audio based on a preset sampling frequency to obtain an audio parameter corresponding to each of audio frames after sampling; and

generating, for each of the audio frames after sampling, a spectrogram corresponding to the audio frame after sampling by performing Fourier transform on the audio parameter corresponding to the audio frame after sampling.

3. The method according to claim 2 , wherein the generating, for each of the audio frames after sampling, a spectrogram corresponding to the audio frame after sampling comprises:

determining, for each of the audio frames, an amplitude of the audio frame;

determining, for each of the audio frames, a spectrum envelop corresponding to the spectrogram based on the amplitude of the audio frame, to obtain a plurality of spectrum envelopes; and

for each of the plurality of spectrum envelopes, combining the spectrum envelope with the spectrum corresponding to the spectrum envelop, to obtain a plurality of combined spectrograms.

4. The method according to claim 1 , wherein the generating, based on the generated spectrogram and the target image, a plurality of video frame images, wherein each of the plurality of video frame images corresponds to one of the audio frames and comprises the spectrum corresponding to the one of the audio frames comprises:

blurring the target image to obtain a blurred target image;

obtaining a target region of the target image to obtain a target region image;

combining the target region image with the spectrogram of each of the audio frames to obtain a plurality of combined images; and

using each of the plurality of combined images as a foreground and the blurred target image as a background to generate a plurality of video frame images each comprising the spectrogram.

5. The method according to claim 4 , wherein the obtaining a target region of the target image to obtain a target region image comprises:

determining a region that is in the target image and that corresponds to a target object in the target image; and

obtaining, based on the determined region, a region that comprises the target object and that has a target shape, as the target region image.

6. The method according to claim 4 , wherein before the combining the target region image with the spectrogram of each of the audio frames, the method further comprises:

performing color feature extraction on the blurred target image to obtain color features of respective pixels of the blurred target image;

performing a weighted average on the color features of the respective pixels to determine a color of the blurred target image; and

setting the determined color of the blurred image as a color of the spectrogram.

7. The method according to claim 4 , wherein the spectrogram is a spectral histogram, and the combining the target region image with the spectrogram of each of the audio frames to obtain a plurality of combined images comprises:

arranging the spectral histogram around the target region image to form the plurality of combined images, wherein a height of a spectral column in the spectral histogram represents an amplitude of a corresponding one of the audio frames, and an angle of the spectral column in the spectral histogram relative to an edge of the target region image represents a frequency of the corresponding audio frame.

8. The method according to claim 4 , wherein the using each of the plurality of combined images as a foreground and the blurred target image as a background to generate a plurality of video frame images each comprising the spectrogram comprises:

obtaining a relative positional relationship between the foreground and the background in one of the plurality of video frame images corresponding to an adjacent audio frame of the target audio frame; and

generating, based on the obtained relative positional relationship, a video frame image corresponding to the target audio frame, where a presentation position of the foreground in the video frame image corresponding to the target audio frame is rotated by a present angle relative to a representation position of the foreground in the video frame image corresponding to the adjacent audio frame.

9. An apparatus for generating a video file, comprising:

at least one processor; and

at least one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor cause the apparatus to perform operations comprising:

presenting, in response to a received video editing instruction, a video editing interface, wherein the video editing interface is configured to enable a selection of an image and a selection of an audio;

determining a target audio and a target image in response to receiving input via the video editing interface;

obtaining, for each of audio frames of the target audio, an audio parameter corresponding to the audio frame;

generating, for each of the audio frames, a spectrogram corresponding to the audio frame based on the obtained audio parameter;

generating, based on the generated spectrogram and the target image, a plurality of video frame images, wherein each of the plurality of video frame images corresponds to one of the audio frames, and each of the plurality of video frame images comprises an image of the spectrum corresponding to the one of the audio frames; and

performing, based on the plurality of video frame images and the target audio, video encoding to generate a target video file.

10. The apparatus according to claim 9 ,

wherein the generating, for each of the audio frames, a spectrogram corresponding to the audio frame based on the obtained audio parameter comprises:

sampling the target audio based on a preset sampling frequency to obtain an audio parameter corresponding to each of audio frames after sampling; and

generating, for each of the audio frames after sampling, a spectrogram corresponding to the audio frame after sampling by performing Fourier transform on the audio parameter corresponding to the audio frame after sampling.

11. The apparatus according to claim 9 ,

wherein the generating, for each of the audio frames after sampling, a spectrogram corresponding to the audio frame after sampling comprises:

determining, for each of the audio frames, an amplitude of the audio frame;

determining, for each of the audio frames, a spectrum envelop corresponding to the spectrogram based on the amplitude of the audio frame, to obtain a plurality of spectrum envelopes; and

for each of the plurality of spectrum envelopes, combining the spectrum envelope with the spectrum corresponding to the spectrum envelop, to obtain a plurality of combined spectrograms.

12. The apparatus according to claim 9 ,

wherein the generating, based on the generated spectrogram and the target image, a plurality of video frame images, wherein each of the plurality of video frame images corresponds to one of the audio frames and comprises the spectrum corresponding to the one of the audio frames comprises:

blurring the target image to obtain a blurred target image;

obtaining a target region of the target image to obtain a target region image;

combining the target region image with the spectrogram of each of the audio frames to obtain a plurality of combined images; and

using each of the plurality of combined images as a foreground and the blurred target image as a background to generate a plurality of video frame images each comprising the spectrogram.

13. The apparatus according to claim 12 ,

wherein the obtaining a target region of the target image to obtain a target region image comprises:

determining a region that is in the target image and that corresponds to a target object in the target image; and

obtaining, based on the determined region, a region that comprises the target object and that has a target shape, as the target region image.

14. The apparatus according to claim 12 ,

wherein before the combining the target region image with the spectrogram of each of the audio frames, the at least one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor also cause the apparatus to:

performing color feature extraction on the blurred target image to obtain color features of respective pixels of the blurred target image;

performing a weighted average on the color features of the respective pixels to determine a color of the blurred target image; and

setting the determined color of the blurred image as a color of the spectrogram.

15. The apparatus according to claim 12 ,

wherein the spectrogram is a spectral histogram, and the combining the target region image with the spectrogram of each of the audio frames to obtain a plurality of combined images comprises:

arranging the spectral histogram around the target region image to form the plurality of combined images, wherein a height of a spectral column in the spectral histogram represents an amplitude of a corresponding one of the audio frames, and an angle of the spectral column in the spectral histogram relative to an edge of the target region image represents a frequency of the corresponding audio frame.

16. The apparatus according to claim 12 ,

wherein the using each of the plurality of combined images as a foreground and the blurred target image as a background to generate a plurality of video frame images each comprising the spectrogram comprises:

obtaining a relative positional relationship between the foreground and the background in one of the plurality of video frame images corresponding to an adjacent audio frame of the target audio frame; and

generating, based on the obtained relative positional relationship, a video frame image corresponding to the target audio frame, where a presentation position of the foreground in the video frame image corresponding to the target audio frame is rotated by a present angle relative to a representation position of the foreground in the video frame image corresponding to the adjacent audio frame.

17. A non-transitory storage medium bearing computer-readable instructions that upon execution on a computing device cause the computing device at least to perform operations comprising:

presenting, in response to a received video editing instruction, a video editing interface, wherein the video editing interface is configured to enable a selection of an image and a selection of an audio;

determining a target audio and a target image in response to receiving input via the video editing interface;

obtaining, for each of audio frames of the target audio, an audio parameter corresponding to the audio frame;

generating, for each of the audio frames, a spectrogram corresponding to the audio frame based on the obtained audio parameter;

generating, based on the generated spectrogram and the target image, a plurality of video frame images, wherein each of the plurality of video frame images corresponds to one of the audio frames, and each of the plurality of video frame images comprises an image of the spectrum corresponding to the one of the audio frames; and

performing, based on the plurality of video frame images and the target audio, video encoding to generate a target video file.

Assignments (4)

Priority Claims (1)

Continuity (1)

References Cited (81)