Systems and methods for processing mixed media streams

1. A computer-implemented method comprising:

receiving, at a server device, a first media stream comprising first media data from a first media object, wherein the first media data was captured by one of:

the server device; or

a first client device;

receiving, at the server device, a second media stream comprising second media data from a second media object, wherein the second media data was captured by one of:

the server device;

the first client device; or

a second client device;

mixing, at the server device, the first media data and the second media data into a third media stream;

compiling, while mixing the third media stream, a metadata stream comprising information enabling:

client-side unmixing of the first media data and the second media data from the third media stream; and

client-side processing of the first media data separate from the second media data;

transmitting, from the server device to one or more additional devices, the third media stream; and

transmitting, from the server device to the one or more additional devices, the metadata stream to enable each of the one or more additional devices to:

unmix the first media data and the second media data from the third media stream; and

process the first media data separate from the second media data.

2. The computer-implemented method of claim 1 , further comprising:

receiving, at the one or more additional devices, the third media stream;

receiving, at the one or more additional devices, the metadata stream;

using, at the one or more additional devices, the metadata stream to unmix the first media data and the second media data from the third media stream; and

differently processing, at the one or more additional devices, the first media data and the second media data.

3. The computer-implemented method of claim 2 , wherein differently processing the first media data and the second media data comprises spatially localizing, at the one or more additional devices while presenting the first media data and the second media data to a user of the one or more additional devices, the first media data and the second media data at different spatial locations in the user's auditory field.

4. The computer-implemented method of claim 3 , wherein:

the one or more additional devices comprise a head-mounted device capable of measuring a head pose of the user; and

the first media data and the second media data are spatially localized relative to the head pose of the user.

5. The computer-implemented method of claim 2 , wherein differently processing the first media data and the second media data comprises independently adjusting, at the one or more additional devices, a volume level of the first media data and a volume level of the second media data.

6. The computer-implemented method of claim 1 , wherein:

the first media stream is received from the first client device;

the first media object is a user of the first client device;

the second media stream is received from the second client device;

the second media object is a user of the second client device;

the server device is a cloud-based server hosting a virtualized conference for a user of the one or more additional devices, the user of the first client device, and the user of the second client device; and

the computer-implemented method further comprises:

capturing, at the first client device, the first media data from the user of the first client device; and

capturing, at the second client device, the second media data from the user of the second client device.

7. The computer-implemented method of claim 1 , further comprising:

receiving, at the server device, a fourth media stream comprising third media data from a third media object;

using a psychoacoustic model to predict that a user of the one or more additional devices would be unable to perceive the third media data if presented to the user of the one or more additional devices; and

refraining, at the server device, from mixing the third media data into the third media stream.

8. The computer-implemented method of claim 1 , wherein:

the one or more additional devices comprise a third client device and a fourth client device; and

the computer-implemented method further comprises:

receiving, at the third client device, the third media stream and the metadata stream;

receiving, at the fourth client device, the third media stream and the metadata stream;

using, at each of the third client device and the fourth client device, the metadata stream to unmix the first media data and the second media data from the third media stream; and

performing, at the third client device, a first operation on the first media data but not the second media data;

performing, at the fourth client device, a second operation on the first media data but not the second media data, wherein the first operation and the second operation are different operations.

9. The computer-implemented method of claim 1 , wherein:

the metadata stream further comprises at least one label of the first media data and at least one label of the second media data;

the server device comprises a sensor array capable of spatial selectivity;

receiving the first media stream comprises capturing, by the server device via the sensor array, the first media data from a first direction in a sound field;

receiving the second media stream comprises capturing, by the server device via the sensor array, the second media data from a second direction in the sound field;

the at least one label of the first media data comprises the first direction; and

the at least one label of the second media data comprises the second direction.

10. The computer-implemented method of claim 1 , wherein:

the metadata stream further comprises at least one label of the first media data and at least one label of the second media data;

the first client device comprises a simultaneous mapping and localization subsystem configured to map an environment of the first client device and localize the first client device within the environment;

receiving the first media stream comprises receiving the first media stream from the first client device;

receiving the second media stream comprises receiving the second media stream from the first client device;

the computer-implemented method further comprises:

capturing, by the first client device, the first media data from a first object in the environment; and

capturing, by the first client device, the second media data from a second object in the environment;

the at least one label of the first media data comprises an attribute of the first object; and

the at least one label of the second media data comprises an attribute of the second object.

11. The computer-implemented method of claim 1 , wherein:

mixing the first media data and the second media data into the third media stream comprises performing a convolution operation on the first media data and the second media data to produce the third media stream;

the information enabling unmixing of the first media data and the second media data from the third media stream comprises information enabling, at each of the one or more additional devices, a deconvolution operation to be performed on the third media stream to produce the first media data and the second media data; and

at least one of the one or more additional devices unmixes the first media data and the second media data from the third media stream by performing the deconvolution operation on the third media stream.

12. A computer-implemented method comprising:

receiving, at one or more client devices from a server device, a media stream comprising first media data from a first media object and second media data from a second media object, wherein:

the server device mixed the first media data and the second media data into the media stream;

the first media data was captured by one of:

the server device; or

a first additional client device; and

the second media data was captured by one of:

the server device;

the first additional client device; or

a second additional client device;

receiving, at the one or more client devices from the server device, a metadata stream comprising information enabling:

client-side extraction of the first media data and the second media data from the media stream; and

client-side processing of the first media data separate from the second media data;

using, at each of the one or more client devices, the metadata stream to extract the first media data and the second media data from the media stream;

differently processing, at each of the one or more client devices, the first media data and the second media data; and

presenting, to a user of each of the one or more client devices, the first media data or the second media data.

13. The computer-implemented method of claim 12 , wherein differently processing the first media data and the second media data comprises spatially localizing, at the one or more client devices before presenting the first media data and the second media data, the first media data and the second media data at different spatial locations in the user's auditory field.

14. The computer-implemented method of claim 13 , further comprising measuring a head pose of the user, wherein the first media data and the second media data are spatially localized relative to the head pose of the user.

15. The computer-implemented method of claim 12 , wherein differently processing the first media data and the second media data comprises independently adjusting, at the one or more client devices, a volume level of the first media data and a volume level of the second media data.

16. A system comprising:

at least one physical processor; and

physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

receive, at a server device, a first media stream comprising first media data from a first media object, wherein the first media data was captured by one of:

the server device; or

a first client device;

receive, at the server device, a second media stream comprising second media data from a second media object, wherein the second media data was captured by one of:

the server device;

the first client device; or

a second client device;

mix, at the server device, the first media data and the second media data into a third media stream;

compile, while mixing the third media stream, a metadata stream comprising information enabling:

client-side unmixing of the first media data and the second media data from the third media stream; and

client-side processing of the first media data separate from the second media data;

transmit, from the server device to one or more additional devices, the third media stream; and

transmit, from the server device to the one or more additional devices, the metadata stream to enable each of the one or more additional devices to:

unmix the first media data and the second media data from the third media stream; and

process the first media data separate from the second media data.

17. The system of claim 16 , wherein:

the first media stream is received from the first client device;

the first media object is a user of the first client device;

the second media stream is received from the second client device;

the second media object is a user of the second client device; and

the server device is a cloud-based server hosting a virtualized conference for a user of the one or more additional devices, the user of the first client device, and the user of the second client device.

18. The system of claim 16 , wherein the physical memory further comprises additional computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

receive, at the server device, a fourth media stream comprising third media data from a third media object;

use a psychoacoustic model to predict that a user of the one or more additional devices would be unable to perceive the third media data if presented to the user of the one or more additional devices; and

refrain, at the server device, from mixing the third media data into the third media stream.

19. The system of claim 16 , wherein:

the metadata stream further comprises at least one label of the first media data and at least one label of the second media data;

the server device comprises a sensor array capable of spatial selectivity;

the sensor array receives the first media stream from a first direction in a sound field;

the sensor array receives the second media stream from a second direction in the sound field;

the at least one label of the first media data comprises the first direction; and

the at least one label of the second media data comprises the second direction.

20. The system of claim 16 , wherein:

the metadata stream further comprises at least one label of the first media data and at least one label of the second media data;

the first client device comprises a simultaneous mapping and localization subsystem configured to map an environment of the first client device and localize the first client device within the environment;

the first media data is captured by the first client device from a first object in the environment;

the second media data is captured by the first client device from a second object in the environment;

the at least one label of the first media data comprises an attribute of the first object; and

the at least one label of the second media data comprises an attribute of the second object.