Granted Patent B2

US 12,256,169 · App. 18/407,825 · Granted Mar 18, 2025

Inventors: Hiroyuki Honma (Chiba, JP); Yuki Yamamoto (Tokyo, JP)

Assignee: Sony Group Corporation

H04N5/9202G06V20/46G06V40/16G06V40/161G10L19/00G10L19/008G10L21/0272G11B27/3081H04N9/802H04N19/46H04R1/40H04R3/00G06F2218/22

Abstract

Claims (41)

1. A video-audio processing apparatus, comprising:

processing circuitry and a memory containing instructions that, when executed by the processing circuitry, are configured to:

cause one or more video objects, based on a video signal, to be displayed in an image;

select a video object from the one or more video objects;

extract an audio object signal of the selected video object from an audio signal; and

produce metadata of the selected video object, the metadata including spread information indicating a spatial spread of an area of the selected video object, wherein the audio object signal of the selected video object is reproduced based on the spatial spread of the selected video object,

wherein the spread information is produced based on a frame image surrounding the selected video object, and

wherein the spread information is produced based on an angle between a first vector from an origin to a center of the frame image and a second vector from the origin to a side of the frame image.

2. The video-audio processing apparatus according to claim 1 , wherein

the processing circuitry is configured to extract a signal other than the audio object signal of the selected video object as a background sound signal from the audio signal.

3. The video-audio processing apparatus according to claim 1 , wherein

the processing circuitry is configured to produce object position information indicating a position, on a space, of the selected video object, and to extract the audio object signal based on the object position information.

4. The video-audio processing apparatus according to claim 3 , wherein

the processing circuitry is configured to extract the audio object signal through sound source separation using the object position information.

5. The video-audio processing apparatus according to claim 4 , wherein

the processing circuitry is configured to carry out fixed beam forming as the sound source separation.

6. The video-audio processing apparatus according to claim 1 , wherein the processing circuitry is further configured to recognize the video object based on the video signal, and to cause an image based on a recognition result of the video object to be displayed together with the video object.

7. The video-audio processing apparatus according to claim 6 , wherein

the processing circuitry is configured to recognize the video object by face recognition.

8. The video-audio processing apparatus according to claim 1 , wherein

the processing circuitry is configured to include object position information indicating a position, on a space, of the selected video object in the metadata.

9. The video-audio processing apparatus according to claim 1 , wherein

the processing circuitry is configured to include a processing priority of the selected video object in the metadata.

10. The video-audio processing apparatus according to claim 1 , wherein the processing circuitry is further configured to encode the audio object signal and the metadata.

11. The video-audio processing apparatus according to claim 10 , wherein the processing circuitry is further configured to encode the video signal, and to multiplex a video bit stream obtained by encoding the video signal, and an audio bit stream obtained by encoding the audio object signal and the metadata.

12. The video-audio processing apparatus according to claim 1 , further comprising an image pickup device configured to obtain the video signal by photographing.

13. The video-audio processing apparatus according to claim 1 , further comprising a sound acquisition device configured to obtain the audio signal by sound acquisition.

14. A video-audio processing method executed by processing circuitry, the method comprising:

causing one or more video objects, based on a video signal, to be displayed in an image;

selecting a video object from the one or more video objects;

extracting an audio object signal of the selected video object from an audio signal; and

producing metadata of the selected video object, the metadata including spread information indicating a spatial spread of an area of the selected video object, wherein the audio object signal of the selected video object is reproduced based on the spatial spread of the selected video object,

wherein the spread information is produced based on a frame image surrounding the selected video object, and

wherein the spread information is produced based on an angle between a first vector from an origin to a center of the frame image and a second vector from the origin to a side of the frame image.

15. A non-transitory computer readable medium containing instructions that, when executed by processing circuitry, perform a video-audio processing method comprising:

causing one or more video objects, based on a video signal, to be displayed in an image;

selecting a video object from the one or more video objects;

extracting an audio object signal of the selected video object from an audio signal; and

producing metadata of the selected video object, the metadata including spread information indicating a spatial spread of an area of the selected video object, wherein the audio object signal of the selected video object is reproduced based on the spatial spread of the selected video object,

wherein the spread information is produced based on a frame image surrounding the selected video object, and

wherein the spread information is produced based on an angle between a first vector from an origin to a center of the frame image and a second vector from the origin to a side of the frame image.

Assignments (2)

Priority Claims (1)

Continuity (3)

References Cited (116)