Sound Source association
Multiple Holocam Orbs observe a real-life environment and generate an artificial reality representation of the real-life environment. Depth image data is cleansed of error due to LED shadow by identifying the edge of a foreground object in an (near infrared light) intensity image, identifying an edge in a depth image, and taking the difference between the start of both edges. Depth data error due to parallax is identified noting when associated text data in a given pixel row that is progressing in a given row direction (left-to-right or right-to-left) reverses order. Sound sources are identified by comparing results of a blind audio source localization algorithm, with the spatial 3D model provided by the Holocam Orb. Sound sources that corresponding to identifying 3D objects are associated together. Additionally, types of data supported by a standard movie data container, such as an MPEG container, is expanding to incorporate free viewpoint data (FVD) model data. This is done by inserting FVD data of different individual 3D objects at different sample rates into a single video stream. Each 3D object is separately identified by a separately assigned ID.
1. A sound location system comprising:
an array of microphones that capture sound of a three-dimensional (3D), real-world scene;
a 3D polyhedron model of the real-world scene, the 3D polyhedron model identifying at least one independent 3D model fragment corresponding to an object within the real-world scene;
a sound source separation module that identifies separated sound sources within the capture sound, and provides at least one theoretical position in 3D space associated with each separated sound source;
a sound source localization module that, for each separate sound source, compares its associated theoretical positions with the positions of each 3D model fragment, identifies a theoretical position in 3D space that agrees with a position of a 3D model fragment, and localizes the position of the sound source with the 3D model fragment whose position agrees with the theoretical position in 3D space.
2. The sound location system of claim 1 , wherein the theoretical position in 3D space, compares, in turn, each theoretical position of each separated sound source with the positions of the 3D model fragments, and during a turn, upon identifying a theoretical position of a separated sound that agrees with a position of a 3D model fragment, ceases to comparing any remaining theoretical positions associated with the separated sound.
3. The sound location system of claim 1 , wherein a theoretical position of a separated sound source is identified as agreeing with the position of a 3D model fragment if the theoretical position is located within a 3D space defined by the 3D model fragment.
4. The sound location system of claim 1 , wherein a theoretical position of a separated sound source is identified as agreeing with the position of a 3D model fragment if the theoretical position is located within a predefined distance from a 3D space defined by the 3D model fragment.
5. The sound location system of claim 4 , wherein the predefined distance is defined as a fraction less than unity of a distance from a centroid of a 3D model to a surface of the 3D model fragment along a direction radially out from the centroid to the theoretical position of the separated sound source.
6. The sound location system of claim 1 , wherein a theoretical position of a separated sound source is identified as not agreeing with a position of a 3D model fragment if the theoretical position is not located within a predefined distance from a 3D space defined by the 3D model fragment.
7. The sound location system of claim 1 , wherein in response to a separated sound source not having an associated theoretical position that agrees with a position of any of the 3D model fragments, all theoretical locations associated with the sound source are discarded.
8. The sound location system of claim 1 , wherein a separated sound source has a plurality of theoretical positions in 3D space.
9. A method of identifying a sound source in an augmented reality reconstruction of a three-dimensional (3D), real-world scene, comprising:
using a microphone array to capture sound of the real-world scene;
accessing a polyhedron model of the real-world scene;
subjecting the captured sound to a blind audio source location algorithm that identifies separated sound sources within the capture sound, and provides at least one theoretical position in 3D space associated with each separated sound source;
for each separate sound source, comparing its associated theoretical positions with the positions of each 3D model fragment;
identifying a theoretical position in 3D space that agrees with a position of any of the 3D model fragments;
localizing the position of the sound source with the 3D model fragment whose position agrees with its associated theoretical position in 3D space.
10. The method of claim 9 , wherein the theoretical position in 3D space, compares, in turn, each theoretical position of each separated sound source with the positions of the 3D model fragments, and during a turn, upon identifying a theoretical position of a separated sound that agrees with a position of a 3D model fragment, ceases to comparing any remaining theoretical positions associated with the separated sound.
11. The method of claim 9 , wherein a theoretical position of a separated sound source is identified as agreeing with the position of a 3D model fragment if the theoretical position is located within a 3D space defined by the 3D model fragment.
12. The method of claim 9 , wherein a theoretical position of a separated sound source is identified as agreeing with the position of a 3D model fragment if the theoretical position is located within a predefined distance from a 3D space defined by the 3D model fragment.
13. The method of claim 12 , wherein the predefined distance is defined as a fraction less than unity of a distance from a centroid of a 3D model to a surface of the 3D model fragment along a direction radially out from the centroid to the theoretical position of the separated sound source.
14. The method of claim 9 , wherein a theoretical position of a separated sound source is identified as not agreeing with a position of a 3D model fragment if the theoretical position is not located within a predefined distance from a 3D space defined by the 3D model fragment.
15. The method of claim 9 , wherein in response to a separated sound source not having an associated theoretical position that agrees with a position of any of the 3D model fragments, all theoretical locations associated with the sound source are discarded.
16. The method of claim 9 , wherein a separated sound source has a plurality of theoretical positions in 3D space.