Granted Patent B2

US 12,505,826 · App. 18/203,469 · Granted Dec 23, 2025

Inventors: Binghuai Lin (Shenzhen, CN); Liyuan Wang (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G10L15/02G10L15/16G10L15/22G10L2015/025G10L15/30

Abstract

Claims (68)

1 . An audio processing method performed by an electronic device, the method comprising:

determining a phoneme feature of at least one phoneme of a given text;

determining an audio feature of an audio frame in audio data corresponding to the given text;

for the audio frame:

obtaining a weight of the phoneme feature of the at least one phoneme based on a mapping relationship between the phoneme feature of the at least one phoneme and the audio feature of the audio frame;

fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame; and

determining a start time and a stop time of a phoneme in the audio data based on the fused feature of the audio frame.

2 . The method according to claim 1 , wherein a phoneme corresponding to an audio frame in the audio data is determined based on the fused feature of the audio frame.

3 . The method according to claim 1 , wherein the determining a phoneme feature of at least one phoneme of a given text comprises:

for each phoneme:

determining a characteristic representation feature of the phoneme;

determining a location representation feature of the phoneme representing a location of the phoneme in a corresponding text unit; and

adding the location representation feature and the characteristic representation feature to obtain the phoneme feature of the phoneme.

4 . The method according to claim 1 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame comprises:

performing value vector transformation on the phoneme feature of the at least one phoneme to obtain a value vector;

multiplying the weight of the phoneme feature of the at least one phoneme with the value vector to obtain an attention result corresponding to the at least one phoneme; and

fusing the attention result corresponding to the at least one phoneme and the audio feature of the audio frame to obtain the fused feature corresponding to the audio frame.

5 . The method according to claim 1 , wherein the determining a start time and a stop time of a phoneme in the audio data based on the fused feature of the audio frame comprises:

determining at least one audio frame corresponding to the phoneme;

determining a start time and a stop time of consecutive audio frames corresponding to the phoneme as the start time and stop time of the phoneme when the phoneme corresponds to a plurality of consecutive audio frames; and

determining a start time and a stop time of the audio frame corresponding to the phoneme as the start time and stop time of the phoneme in the audio data when the phoneme corresponds to one audio frame.

6 . The method according to claim 1 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame is implemented by invoking an attention fusion network, the determination of a phoneme corresponding to each audio frame is implemented by invoking a phoneme classification network.

7 . An electronic device, comprising:

a memory, configured to store a computer-executable instruction; and

a processor, configured to implement a audio processing method by executing the computer-executable instruction stored in the memory, the method including:

determining a phoneme feature of at least one phoneme of a given text;

determining an audio feature of an audio frame in audio data corresponding to the given text;

for the audio frame:

obtaining a weight of the phoneme feature of the at least one phoneme based on a mapping relationship between the phoneme feature of the at least one phoneme and the audio feature of the audio frame;

fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame; and

determining a start time and a stop time of a phoneme in the audio data based on the fused feature of the audio frame.

8 . The electronic device according to claim 7 , wherein a phoneme corresponding to an audio frame in the audio data is determined based on the fused feature of the audio frame.

9 . The electronic device according to claim 7 , wherein the determining a phoneme feature of at least one phoneme of a given text comprises:

for each phoneme:

determining a characteristic representation feature of the phoneme;

determining a location representation feature of the phoneme representing a location of the phoneme in a corresponding text unit; and

adding the location representation feature and the characteristic representation feature to obtain the phoneme feature of the phoneme.

10 . The electronic device according to claim 7 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame comprises:

performing value vector transformation on the phoneme feature of the at least one phoneme to obtain a value vector;

multiplying the weight of the phoneme feature of the at least one phoneme with the value vector to obtain an attention result corresponding to the at least one phoneme; and

fusing the attention result corresponding to the at least one phoneme and the audio feature of the audio frame to obtain the fused feature corresponding to the audio frame.

11 . The electronic device according to claim 7 , wherein the determining a start time and a stop time of each phoneme in the audio data based on the fused feature of the audio frame comprises:

determining at least one audio frame corresponding to the phoneme;

determining a start time and a stop time of consecutive audio frames corresponding to the phoneme as the start time and stop time of the phoneme when the phoneme corresponds to a plurality of consecutive audio frames; and

determining a start time and a stop time of the audio frame corresponding to the phoneme as the start time and stop time of the phoneme in the audio data when the phoneme corresponds to one audio frame.

12 . The electronic device according to claim 7 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame is implemented by invoking an attention fusion network, the determination of a phoneme corresponding to each audio frame is implemented by invoking a phoneme classification network.

13 . A non-transitory computer-readable storage medium, storing a computer-executable instruction that, when executed by a processor of an electronic device, causing the electronic device to implement an audio processing method including:

determining a phoneme feature of at least one phoneme of a given text;

determining an audio feature of an audio frame in audio data corresponding to the given text;

for the audio frame:

obtaining a weight of the phoneme feature of the at least one phoneme based on a mapping relationship between the phoneme feature of the at least one phoneme and the audio feature of the audio frame;

fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame; and

determining a start time and a stop time of a phoneme in the audio data based on the fused feature of the audio frame.

14 . The non-transitory computer-readable storage medium according to claim 13 , wherein a phoneme corresponding to an audio frame in the audio data is determined based on the fused feature of the audio frame.

15 . The non-transitory computer-readable storage medium according to claim 13 , wherein the determining a phoneme feature of at least one phoneme of a given text comprises:

for each phoneme:

determining a characteristic representation feature of the phoneme;

determining a location representation feature of the phoneme representing a location of the phoneme in a corresponding text unit; and

adding the location representation feature and the characteristic representation feature to obtain the phoneme feature of the phoneme.

16 . The non-transitory computer-readable storage medium according to claim 13 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame comprises:

performing value vector transformation on the phoneme feature of the at least one phoneme to obtain a value vector;

multiplying the weight of the phoneme feature of the at least one phoneme with the value vector to obtain an attention result corresponding to the at least one phoneme; and

fusing the attention result corresponding to the at least one phoneme and the audio feature of the audio frame to obtain the fused feature corresponding to the audio frame.

17 . The non-transitory computer-readable storage medium according to claim 13 , wherein the determining a start time and a stop time of each phoneme in the audio data based on the fused feature of the audio frame comprises:

determining at least one audio frame corresponding to the phoneme;

determining a start time and a stop time of consecutive audio frames corresponding to the phoneme as the start time and stop time of the phoneme when the phoneme corresponds to a plurality of consecutive audio frames; and

determining a start time and a stop time of the audio frame corresponding to the phoneme as the start time and stop time of the phoneme in the audio data when the phoneme corresponds to one audio frame.

18 . The non-transitory computer-readable storage medium according to claim 13 , wherein the fusing the audio feature of the audio frame and the phoneme feature of the at least one phoneme based on the weight of the phoneme feature of the at least one phoneme to obtain a fused feature of the audio frame is implemented by invoking an attention fusion network, the determination of a phoneme corresponding to each audio frame is implemented by invoking a phoneme classification network.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (17)