Granted Patent B2

US 12,444,191 · App. 18/133,641 · Granted Oct 14, 2025

Inventor: Yi Zhang (Shanghai, CN)

Assignee: Shanghai Hode Information Technology Co., Ltd.

G06V20/41G06F16/7834G06V20/46

Abstract

Claims (77)

1. A method, comprising:

performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map, and extracting a semantic segmentation feature of the to-be-processed video data based on the semantic segmentation map;

extracting an audio feature of each audio file in a pre-established audio set; and

aligning the audio feature and the semantic segmentation feature, selecting a target audio file from the audio set based on an alignment result, and constructing background audio for the to-be-processed video data based on the target audio file,

wherein the aligning the audio feature and the semantic segmentation feature comprises:

performing dimension scaling processing on the audio feature and the semantic segmentation feature based on a preset feature dimension, to generate a target audio feature and a target semantic segmentation feature; and

aligning the target audio feature and the target semantic segmentation feature.

2. The method according to claim 1 , wherein the performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map comprises:

dividing the to-be-processed video data into video segments based on a preset duration threshold;

extracting a first key frame of each first video segment in a division result; and

inputting the first key frame into a semantic segmentation model for processing, to generate a first semantic segmentation map of each first video segment.

3. The method according to claim 2 , wherein the extracting a semantic segmentation feature of the to-be-processed video data based on the semantic segmentation map comprises:

extracting a first semantic segmentation feature of each first video segment based on the first semantic segmentation map; and

calculating an average value of first semantic segmentation features of all the first video segments in the division result, and using the average value as the semantic segmentation feature of the to-be-processed video data.

4. The method according to claim 2 , wherein the semantic segmentation model is trained as follows:

dividing a sample video file into video segments based on the preset duration threshold;

extracting a second key frame of each second video segment in a division result; and

using the second key frame as sample data, using a category identifier of each pixel in a semantic segmentation map of the second key frame as a label, and inputting them to a to-be-trained semantic segmentation model for training, to obtain the semantic segmentation model, wherein the semantic segmentation model associates the second key frame with the category identifier of each pixel.

5. The method according to claim 1 , wherein the extracting an audio feature of each audio file in a pre-established audio set comprises:

dividing each audio file in the audio set based on the preset duration threshold;

performing Fourier transform on each first audio segment in a division result, to generate a first spectrum signal of each first audio segment; and

inputting the first spectrum signal into an audio feature extraction model for processing, to generate the audio feature of each audio file in the audio set.

6. The method according to claim 5 , wherein the audio feature extraction model is trained as follows:

dividing a sample audio file based on the preset duration threshold;

performing Fourier transform on each second audio segment in a division result, to generate a second spectrum signal of each second audio segment; and

using the second spectrum signal as sample data, using an audio type of the sample audio file as a label, and inputting them to a to-be-trained audio feature extraction model for training, to obtain the audio feature extraction model, wherein the audio feature extraction model associates the second spectrum signal with the audio type.

7. The method according to claim 1 , wherein the aligning the audio feature and the semantic segmentation feature, selecting a target audio file from the audio set based on an alignment result, and constructing background audio for the to-be-processed video data based on the target audio file comprises:

calculating a distance between the audio feature and the semantic segmentation feature; and

using an audio file corresponding to an audio feature whose distance from the semantic segmentation feature is less than a preset distance threshold as the target audio file, and constructing the background audio based on the target audio file.

8. The method according to claim 1 , wherein the aligning the audio feature and the semantic segmentation feature comprises:

inputting the audio feature and the semantic segmentation feature into an audio alignment model for alignment.

9. The method according to claim 8 , wherein the audio alignment model is trained as follows:

constructing a plurality of triplet training samples comprising sample video data, positive sample audio data, and negative sample audio data;

inputting the plurality of triplet training samples into the audio alignment model, to obtain feature vectors of sample video data, positive sample audio data, and negative sample audio data in each triplet training sample; and

calculating a first distance between the feature vectors of the sample video data and the positive sample audio data and a second distance between the feature vectors of the sample video data and the negative sample audio data in each triplet training sample, inputting the first distance and the second distance into a metric learning loss function, and training the audio alignment model based on an output of the loss function until the loss function tends to be stable.

10. A computing device, comprising:

a memory and a processor, wherein

the memory is configured to store computer-executable instructions, the processor is configured to execute the computer-executable instructions, when executing the computer-executable instructions, the processor perform operations comprising:

performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map, and extracting a semantic segmentation feature of the to-be-processed video data based on the semantic segmentation map;

extracting an audio feature of each audio file in a pre-established audio set; and

aligning the audio feature and the semantic segmentation feature, selecting a target audio file from the audio set based on an alignment result, and constructing background audio for the to-be-processed video data based on the target audio file,

wherein the aligning the audio feature and the semantic segmentation feature comprises:

performing dimension scaling processing on the audio feature and the semantic segmentation feature based on a preset feature dimension, to generate a target audio feature and a target semantic segmentation feature; and

aligning the target audio feature and the target semantic segmentation feature.

11. The computing device according to claim 10 , wherein the performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map comprises:

dividing the to-be-processed video data into video segments based on a preset duration threshold;

extracting a first key frame of each first video segment in a division result; and

inputting the first key frame into a semantic segmentation model for processing, to generate a first semantic segmentation map of each first video segment.

12. The computing device according to claim 11 , wherein the extracting a semantic segmentation feature of the to-be-processed video data based on the semantic segmentation map comprises:

extracting a first semantic segmentation feature of each first video segment based on the first semantic segmentation map; and

calculating an average value of first semantic segmentation features of all the first video segments in the division result, and using the average value as the semantic segmentation feature of the to-be-processed video data.

13. The computing device according to claim 11 , wherein the semantic segmentation model is trained as follows:

dividing a sample video file into video segments based on the preset duration threshold;

extracting a second key frame of each second video segment in a division result; and

using the second key frame as sample data, using a category identifier of each pixel in a semantic segmentation map of the second key frame as a label, and inputting them to a to-be-trained semantic segmentation model for training, to obtain the semantic segmentation model, wherein the semantic segmentation model associates the second key frame with the category identifier of each pixel.

14. The computing device according to claim 10 , wherein the extracting an audio feature of each audio file in a pre-established audio set comprises:

dividing each audio file in the audio set based on the preset duration threshold;

performing Fourier transform on each first audio segment in a division result, to generate a first spectrum signal of each first audio segment; and

inputting the first spectrum signal into an audio feature extraction model for processing, to generate the audio feature of each audio file in the audio set.

15. The computing device according to claim 14 , wherein the audio feature extraction model is trained as follows:

dividing a sample audio file based on the preset duration threshold;

performing Fourier transform on each second audio segment in a division result, to generate a second spectrum signal of each second audio segment; and

using the second spectrum signal as sample data, using an audio type of the sample audio file as a label, and inputting them to a to-be-trained audio feature extraction model for training, to obtain the audio feature extraction model, wherein the audio feature extraction model associates the second spectrum signal with the audio type.

16. The computing device according to claim 10 , wherein the aligning the audio feature and the semantic segmentation feature, selecting a target audio file from the audio set based on an alignment result, and constructing background audio for the to-be-processed video data based on the target audio file comprises:

calculating a distance between the audio feature and the semantic segmentation feature; and

using an audio file corresponding to an audio feature whose distance from the semantic segmentation feature is less than a preset distance threshold as the target audio file, and constructing the background audio based on the target audio file.

17. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which when executed by a processor, cause the processor to perform operations comprising:

performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map, and extracting a semantic segmentation feature of the to-be-processed video data based on the semantic segmentation map;

extracting an audio feature of each audio file in a pre-established audio set; and

aligning the audio feature and the semantic segmentation feature, selecting a target audio file from the audio set based on an alignment result, and constructing background audio for the to-be-processed video data based on the target audio file,

wherein the aligning the audio feature and the semantic segmentation feature comprises:

performing dimension scaling processing on the audio feature and the semantic segmentation feature based on a preset feature dimension, to generate a target audio feature and a target semantic segmentation feature; and

aligning the target audio feature and the target semantic segmentation feature.

18. The medium according to claim 17 , wherein the performing semantic segmentation on to-be-processed video data to generate a corresponding semantic segmentation map comprises:

dividing the to-be-processed video data into video segments based on a preset duration threshold;

extracting a first key frame of each first video segment in a division result; and

inputting the first key frame into a semantic segmentation model for processing, to generate a first semantic segmentation map of each first video segment.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (20)