Granted Patent B2

US 12,342,153 · App. 17/885,468 · Granted Jun 24, 2025

Inventor: Junjie Shi (Beijing, CN)

Assignee: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.

H04S7/302G06F3/011G06T19/006H04S2400/11

Abstract

Claims (69)

1. An audio processing method, comprising:

acquiring an original image captured by a terminal;

determining a three-dimensional relative position of a target object relative to the terminal as a first three-dimensional relative position according to the original image; and

performing three-dimensional effect processing on a target sound according to the first three-dimensional relative position to enable a sound source position of the target sound in audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to a positional relationship between the target object and a sound effect object corresponding to the target object, wherein the target sound is an effect sound corresponding to the sound effect object;

wherein the sound effect object is in a different position as the target object;

wherein the determining the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the original image comprises:

if the target object does not exist in the original image, acquiring historical motion trajectory information corresponding to the target object, wherein the historical motion trajectory information is obtained by performing motion tracking on an image captured by the terminal before the original image is acquired; and

determining the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the historical motion trajectory information.

2. The method according to claim 1 , wherein the determining the three- dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the original image comprises:

performing feature extraction on the original image to acquire first anchor point information in the original image;

determining, according to pre-stored reference anchor point information corresponding to the target object, whether the first anchor point information is capable of matching the reference anchor point information; and

if it is determined that the first anchor point information matches the reference anchor point information, determining the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the first anchor point information.

3. The method according to claim 1 , wherein the determining the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the original image comprises:

if second anchor point information generated from an anchor point marking operation performed by a user on the original image is acquired, determining the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the second anchor point information.

4. The method according to claim 1 , wherein the performing the three-dimensional effect processing on the target sound according to the first three-dimensional relative position to enable the sound source position of the target sound in the audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object comprises:

determining the three-dimensional relative position of the sound source position of the target sound relative to the terminal as a second three-dimensional relative position according to the first three-dimensional relative position and the positional relationship between the target object and the sound effect object corresponding to the target object; and

performing three-dimensional effect processing on the target sound according to the second three-dimensional relative position to enable a sound source position of the target sound in audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object.

5. The method according to claim 4 , further comprising:

acquiring additional description information of the target sound, wherein the additional description information comprises at least one of a sound source sounding area of the target sound and a sound source environment of the target sound, wherein the sound source environment of the target sound is indoors or outdoors;

the performing the three-dimensional effect processing on the target sound according to the second three-dimensional relative position to enable the sound source position of the target sound in the audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object, comprises:

performing the three-dimensional effect processing on the target sound according to the second three-dimensional relative position and the additional description information to enable the sound source position of the target sound in the audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object.

6. The method according to claim 1 , further comprising:

superimposing image material corresponding to the target object to the first three-dimensional relative position of the original image to generate an augmented reality (AR) image corresponding to the original image; and

displaying the AR image.

7. The method according to claim 1 , further comprising:

determining whether the terminal meets an ambisonic playing condition; and

if it is determined that the terminal meets the ambisonic playing condition, playing the audio obtained after the three-dimensional effect processing.

8. An electronic device, comprising:

a storage apparatus, having a computer program stored thereon; and

a processing apparatus, configured to execute the computer program in the storage apparatus to:

acquire an original image captured by a terminal;

determine a three-dimensional relative position of a target object relative to the terminal as a first three-dimensional relative position according to the original image; and

perform three-dimensional effect processing on a target sound according to the first three-dimensional relative position to enable a sound source position of the target sound in audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to a positional relationship between the target object and a sound effect object corresponding to the target object, wherein the target sound is an effect sound corresponding to the sound effect object;

wherein the sound effect object is in a different position as the target object;

wherein the processing apparatus is configured to execute the computer program in the storage apparatus to:

acquire historical motion trajectory information corresponding to the target object if the target object does not exist in the original image, wherein the historical motion trajectory information is obtained by performing motion tracking on an image captured by the terminal before the original image is acquired; and

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the historical motion trajectory information.

9. The electronic device according to claim 8 , wherein the processing apparatus is configured to execute the computer program in the storage apparatus to:

perform feature extraction on the original image to acquire first anchor point information in the original image;

determine, according to pre-stored reference anchor point information corresponding to the target object, whether the first anchor point information is capable of matching the reference anchor point information; and

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the first anchor point information if it is determined that the first anchor point information matches the reference anchor point information.

10. The electronic device according to claim 8 , wherein the processing apparatus is configured to execute the computer program in the storage apparatus to:

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to second anchor point information if the second anchor point information generated from an anchor point marking operation performed by a user on the original image is acquired.

11. The electronic device according to claim 8 , wherein the processing apparatus is configured to execute the computer program in the storage apparatus to:

determine the three-dimensional relative position of the sound source position of the target sound relative to the terminal as a second three-dimensional relative position according to the first three-dimensional relative position and the positional relationship between the target object and the sound effect object corresponding to the target object; and

perform three-dimensional effect processing on the target sound according to the second three-dimensional relative position to enable a sound source position of the target sound in audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object.

12. The electronic device according to claim 11 , wherein the processing apparatus is further configured to execute the computer program in the storage apparatus to:

acquire additional description information of the target sound, wherein the additional description information comprises at least one of a sound source sounding area of the target sound and a sound source environment of the target sound, wherein the sound source environment of the target sound is indoors or outdoors;

perform the three-dimensional effect processing on the target sound according to the second three-dimensional relative position and the additional description information to enable the sound source position of the target sound in the audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to the positional relationship between the target object and the sound effect object corresponding to the target object.

13. The electronic device according to claim 8 , wherein the processing apparatus is further configured to execute the computer program in the storage apparatus to:

superimpose image material corresponding to the target object to the first three-dimensional relative position of the original image to generate an augmented reality (AR) image corresponding to the original image; and

display the AR image.

14. The electronic device according to claim 8 , wherein the processing apparatus is further configured to execute the computer program in the storage apparatus to:

determine whether the terminal meets an ambisonic playing condition; and

play the audio obtained after the three-dimensional effect processing if it is determined that the terminal meets the ambisonic playing condition.

15. A non-transitory computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processing apparatus, is configured to:

acquire an original image captured by a terminal;

determine a three-dimensional relative position of a target object relative to the terminal as a first three-dimensional relative position according to the original image; and

perform three-dimensional effect processing on a target sound according to the first three-dimensional relative position to enable a sound source position of the target sound in audio obtained after the three-dimensional effect processing and the first three-dimensional relative position to conform to a positional relationship between the target object and a sound effect object corresponding to the target object, wherein the target sound is an effect sound corresponding to the sound effect object;

wherein the sound effect object is in a different position as the target object;

wherein the program. when executed by a processing apparatus, is configured to:

acquire historical motion trajectory information corresponding to the target object if the target object does not exist in the original image, wherein the historical motion trajectory information is obtained by performing motion tracking on an image captured by the terminal before the original image is acquired; and

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the historical motion trajectory information.

16. The non-transitory computer readable storage medium according to claim 15 , wherein the program, when executed by the processing apparatus, is configured to:

perform feature extraction on the original image to acquire first anchor point information in the original image;

determine, according to pre-stored reference anchor point information corresponding to the target object, whether the first anchor point information is capable of matching the reference anchor point information; and

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to the first anchor point information if it is determined that the first anchor point information matches the reference anchor point information.

17. The non-transitory computer readable storage medium according to claim 15 , wherein the program, when executed by the processing apparatus, is configured to:

determine the three-dimensional relative position of the target object relative to the terminal as the first three-dimensional relative position according to second anchor point information if the second anchor point information generated from an anchor point marking operation performed by a user on the original image is acquired.

Assignments (2)

Priority Claims (1)

Continuity (2)

References Cited (48)