Accessible audio intent visualization for interactive environments

1 . A method, comprising:

generating a virtual environment comprising a virtual sound source;

prior to outputting a sound, classifying an intended effect on a user of the sound that is to be output to the user, the sound corresponding to the virtual sound source, a class of the intended effect being selected from (i) understanding, (ii) awareness, (iii) emotion, and (iv) physiological response, wherein classifying the intended effect on the user comprises analyzing metadata associated with the sound and metadata associated with the virtual sound source;

selecting, according to a consistent visual grammar, a type of visual element based on the classified intended effect, the type of visual element being selected from a color, an icon, a word, or a shape;

outputting the sound; and

producing a visual element of the selected type of visual element for view by the user that has at least one visual property selected to correspond to the classified intended effect, the at least one visual property including one or more of brightness, saturation, shape, and scale.

2 . The method of claim 1 , wherein classifying the intended effect further comprises analyzing a sound component of the sound, and the sound component comprises one or more of:

i. rate of attack in a sound envelope;

ii. rate of decay in the sound envelope;

iii. frequency distribution in the sound;

iv. a presence of vocalization in the sound;

v. absolute volume of the sound;

vi. volume of the sound relative to other sounds in the virtual environment; and

vii. duration of the sound.

3 . The method of claim 1 , further comprising:

generating the metadata associated with the sound.

4 . The method of claim 1 , wherein producing the visual element comprises adding at least one of a plurality of predetermined colors to at least part of the virtual environment, the predetermined colors being respectively associated with different intended effects on the user.

5 . The method of claim 1 , wherein producing a visual element comprises adding at least one of a plurality of predetermined icons or words to at least part of the virtual environment, the predetermined icons or words being respectively associated with different intended effects on the user.

6 . The method of claim 1 , further comprising modifying the visual element to change one or more of:

i. brightness;

ii. saturation;

iii. shape; and

iv. scale,

responsive to one or more sound components of the sound.

7 . The method of claim 1 , wherein producing the visual element comprises adding the visual element at a location in the virtual environment corresponding to a location of the virtual sound source.

8 . The method of claim 1 , wherein producing the visual element comprises adding the visual element at a location in the virtual environment corresponding to a direction or a location of the sound relative to a predetermined position within a virtual sound stage.

9 . The method of claim 1 , wherein producing the visual element comprises driving a directional illumination separate to a primary display of the virtual environment.

10 . The method of claim 9 , wherein the directional illumination comprises one or more of:

i. a directional illumination physically surrounding the primary display;

ii. a directional illumination physically surrounding the user; and

iii. a directional illumination located on a handheld controller used by the user.

11 . The method of claim 1 , wherein the visual element is produced as an overlay at a location in the virtual environment corresponding to a direction or location of the sound relative to a predetermined position within a virtual sound stage.

12 . The method of claim 1 , wherein the consistent visual grammar is provided by an operating system or a helper application for consistency across a plurality of different software products.

13 . The method of claim 1 , further comprising suspending production of the visual element during scripted game elements and resuming thereafter.

14 . The method of claim 1 , wherein the metadata associated with the sound is generated by analyzing the sound to identify one or more audio properties, and the generated metadata is used during gameplay to produce the visual element while the corresponding sound is being output.

15 . The method of claim 1 , wherein producing the visual element comprises displaying an onomatopoeia word selected according to the classified intended effect.

16 . A non-transitory, computer-readable storage medium having stored thereon computer executable instructions adapted to cause a computer system to perform a method comprising:

generating a virtual environment comprising a virtual sound source;

prior to outputting a sound, classifying an intended effect on a user of the sound that is to be output to the user, the sound corresponding to the virtual sound source, a class of the intended effect being selected from (i) understanding, (ii) awareness, (iii) emotion, and (iv) physiological response, wherein classifying the intended effect on the user comprises analyzing metadata associated with the sound and metadata associated with the virtual sound source;

selecting, according to a consistent visual grammar, a type of visual element based on the classified intended effect, the type of visual element being selected from a color, an icon, a word, or a shape;

outputting the sound; and

producing a visual element of the selected type of visual element for view by the user that has at least one visual property selected to correspond to the classified intended effect, the at least one visual property including one or more of brightness, saturation, shape, and scale.

17 . An apparatus, comprising:

one or more processors;

one or more computer-readable storage media storing instructions which, when executed by the one or more processors, cause the apparatus to perform operations comprising:

generating a virtual environment comprising a virtual sound source;

prior to outputting a sound, classifying an intended effect on a user of the sound that is to be output to the user, the sound corresponding to the virtual sound source, a class of the intended effect being selected from (i) understanding, (ii) awareness, (iii) emotion, and (iv) physiological response, wherein classifying the intended effect on the user comprises analyzing metadata associated with the sound and metadata associated with the virtual sound source;

selecting, according to a consistent visual grammar, a type of visual element based on the classified intended effect, the type of visual element being selected from a color, an icon, a word, or a shape;

outputting the sound; and

producing a visual element of the selected type of visual element for view by the user that has at least one visual property selected to correspond to the classified intended effect, the at least one visual property including one or more of brightness, saturation, shape, and scale.