Granted Patent B2

US 12,315,055 · App. 17/925,899 · Granted May 27, 2025

Inventor: Tomohiko Gotoh (Tokyo, JP)

Assignee: SONY GROUP CORPORATION

G06T13/205G06F3/165A63F13/215A63F13/52

Abstract

Claims (69)

1. An information processing apparatus, comprising:

a microphone array configured to detect a sound in a real space around a user,

wherein the microphone array includes at least three microphones having a fixed positional relationship; and

a central processing unit (CPU) configured to:

detect a footstep of the user based on the detected sound;

generate, as environmental information of the real space, ground surface information indicating a material of a ground surface in the real space,

wherein the generation of the ground surface information is based on the detected footstep of the user;

control display of a virtual object in a virtual space,

wherein the virtual space is based on the real space; and

control an operation of the virtual object in the virtual space based on the environmental information.

2. The information processing apparatus according to claim 1 , wherein

the CPU is further configured to:

detect, based on the detected sound, a wind noise associated with a wind in the real space; and

generate, as the environmental information, wind information indicating at least one of a direction or an intensity of the wind in the real space, and

the generation of the wind information is based on the detected wind noise.

3. The information processing apparatus according to claim 2 , wherein the CPU is further configured to control the operation of the virtual object based on the wind information, and

the operation of the virtual object expresses reaction of the virtual object to the wind.

4. The information processing apparatus according to claim 3 , wherein the CPU is further configured to:

detect a shielding structure that blocks the wind in one of the real space or the virtual space; and

adjust, based on the shielding structure, the reaction of the virtual object to the wind in the real space.

5. The information processing apparatus according to claim 1 , wherein

the virtual object is a virtual character, and

the CPU is further configured to control a footstep of the virtual character based on the ground surface information.

6. The information processing apparatus according to claim 1 , wherein the CPU is further configured to generate a first map to which the ground surface information is mapped.

7. The information processing apparatus according to claim 6 , wherein

the virtual object is a virtual character, and

the CPU is further configured to change a footstep of the virtual character based on the first map.

8. The information processing apparatus according to claim 6 , wherein the CPU is further configured to:

acquire a second map that corresponds to a segmentation map of the real space,

wherein the segmentation map is based on an image of the real space; and

generate an integrated map by integration of the first map and the second map.

9. The information processing apparatus according to claim 8 , wherein

the virtual object is a virtual character, and

the CPU is further configured to control a footstep of the virtual character based on the integrated map.

10. The information processing apparatus according to claim 1 , wherein

the CPU is further configured to generate, as the environmental information, noise information indicating a noise level in the real space, and

the generation of the noise information is based on the detected sound.

11. The information processing apparatus according to claim 10 , wherein

the virtual object is a virtual character, and

the CPU is further configured to control, based on the noise information, at least one of a volume of utterance, a content of the utterance, or a way of the utterance by the virtual character.

12. The information processing apparatus according to claim 11 , wherein

the noise information indicates a first noise level at a position of the user, and

the CPU is further configured to:

estimate a second noise level based on the first noise level,

wherein the second noise level is at a position of the virtual character; and

control an operation of the virtual character based on the second noise level.

13. The information processing apparatus according to claim 12 , wherein the CPU is further configured to control the operation of the virtual character in a plurality of stages divided corresponding to the second noise level.

14. The information processing apparatus according to claim 1 , wherein

the virtual space is an extended real space, and

the extended real space is based on superimposition of the virtual object on the real space.

15. The information processing apparatus according to claim 1 , wherein the microphone array is in at least one of a first device wearable by the user or a second device carriable by the user.

16. An information processing method, comprising:

detecting, by a microphone array, a sound in a real space around a user,

wherein the microphone array includes at least three microphones having a fixed positional relationship;

detecting, by a central processing unit (CPU), a footstep of the user based on the detected sound;

generating, by the CPU, ground surface information as environmental information of the real space based on the detected footstep of the user,

wherein the ground surface information indicates a material of a ground surface in the real space;

controlling, by the CPU, display of a virtual object in a virtual space,

wherein the virtual space is based on the real space; and

controlling, by the CPU, an operation of the virtual object in the virtual space based on the environmental information.

17. A non-transitory computer-readable medium having stored thereon computer-executable instructions which, when executed by a computer, cause the computer to execute operations, the operations comprising:

acquiring, from a microphone array, a result of a detection of a sound in a real space around a user,

wherein the microphone array includes at least three microphones having a fixed positional relationship;

detecting a footstep of the user based on the result of the detection of the sound;

generating, as environmental information of the real space, ground surface information indicating a material of a ground surface in the real space,

wherein the generation of the ground surface information is based on the detected footstep of the user;

controlling display of a virtual object in a virtual space,

wherein the virtual space is based on the real space; and

controlling an operation of the virtual object in the virtual space based on the environmental information.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (11)