Interactive mixed reality system for a real-world event
Embodiments are directed to systems and methods for automatically customizing a mixed reality immersive, environment scene of a real-world event. Capture devices (e.g., mobile phones) positioned at the real-world event may capture scenes from the event at different points of view. A remote processing unit may generate a mixed reality environment based on a three-dimensional representation of the real-world event from the captured scenes and various virtual elements. A remote user device may display a representation of this mixed reality environment and detect associated input at the device. The processing unit may, in turn, simulate manipulation of the real-world scenes based on the detected input.
1. A system for remotely interacting with a real-world event, comprising:
a group of capture devices, each of the group of capture devices comprising a camera configured to capture scenes of the real-world event;
a position sensor configured to detect a position of the group of capture devices;
a processing unit communicatively coupled with the group of capture devices and the position sensor over the distributed network, the processing unit configured to:
receive, from each of the group of capture devices, the captured scenes;
receive, from the position sensor, the detected position of the group of capture devices;
generate an immersive representation of a physical environment of the real-world event using the captured scenes from at least two capture devices of the group of capture devices and the detected position of the at least two capture devices;
generate a mixed reality environment of the real-world event by modifying the immersive visual representation of the physical environment with virtual elements; and
transmit a signal including information of the mixed reality environment over one or both of the distributed network or a local connection; and
a first user device communicatively coupled with the processing unit, the first user device comprising:
a communication device configured to receive the signal;
a display configured to depict the mixed reality environment based on the information of the signal; and
an input device operatively coupled with the display and configured to detect input in response to a depiction of the mixed reality environment;
wherein the processing unit is further configured to modify the mixed reality environment in response to the input of the input device.
2. The system of claim 1 , wherein the processing unit is further configured to, in response to the input, generate a constructed asset in the mixed reality environment that simulates manipulation of the immersive representation of the physical environment.
3. The system of claim 2 , wherein:
the input device comprises a sensor configured to detect a characteristic of the input; and
the processing unit is further configured to:
modify the constructed asset in a first manner based on the characteristic of the input being a first value; and
modify the constructed asset in a second manner based on the characteristic of the input being a second value.
4. The system of claim 3 , wherein the input is one or more of a touch input, a force input, a gesture, or a voice command.
5. The system of claim 2 , wherein:
the input device is configured to detect a subsequent input in response to a depiction of the constructed asset in the mixed reality environment; and
the processing unit is further configured to modify the constructed asset in response to the subsequent input.
6. The system of claim 1 , wherein:
the processing unit is further configured to determine tactile feedback information based on the mixed reality environment; and
the first user device further comprises a tactile structure configured to produce tactile feedback corresponding to the mixed reality environment.
7. The system of claim 6 , wherein the tactile feedback information is determined in response to the simulated manipulation of the immersive representation of the physical environment in the mixed reality environment.
8. The system of claim 1 , wherein the real-world event is a live event.
9. The system of claim 1 , wherein the real-world event occurs in a programmed time window.
10. The system of claim 1 , wherein:
the system further comprises a database communicatively coupled with the processing unit over the distributed network; and
the database comprises data elements characteristic of one or both of the real-world event or a first user associated with the first user device.
11. The system of claim 10 , wherein:
the data elements include information associated with a physical location of the real-world event; and
the processing unit is further configured to:
retrieve the data elements over the distributed network; and
associate the group of capture devices with the real-world event based on the detected position of the group of capture devices and the physical location of the real-world event.
12. The system of claim 10 , wherein:
the data elements include information associated with participants of the real-world event; and
the virtual elements of the mixed reality environment include a virtual representation of the information associated with the participants.
13. The system of claim 10 , wherein:
the data elements include a profile of the first user of the first user device; and
the processing unit is further configured to customize the virtual elements based on the profile.
14. The system of claim 13 , wherein the virtual elements include a virtual representation of the first user in the mixed reality environment.
15. The system of claim 1 , wherein the position sensor comprises a plurality of position sensors, each of which is associated with a corresponding one of the group of capture devices.
16. The system of claim 1 , wherein:
the system further comprises a second user device communicatively coupled with the processing unit over the distributed network; and
the second user device comprises a display configured to depict the mixed reality environment, as modified by the input of the first user device.
17. The system of claim 16 , wherein:
the second user device comprises an input device operatively coupled with the display and configured to detect input in response to a depiction of the mixed reality environment;
the processing unit is further configured to modify the mixed reality environment in response to the input of the second input device; and
the display of the first user device is configured to depict the mixed reality environment, as modified by the input of the second user device.
18. The system of claim 1 , wherein:
in a first mode, the display of the first user device is configured to depict the mixed reality environment from a point of view of a first of the group of capture devices;
in a second mode, the display of the first user device is configured to depict the mixed reality environment from a point of view of a second of the group of capture devices; and
the processing unit is further configured to alternate between the first mode and the second mode based on the input of the input structure.
19. A method of generating a mixed reality environment of a live event for a remote user, comprising:
accessing information associated with a physical location of the live event from a remote database over a distributed network;
automatically receiving a position of each of a group of candidate live event devices in substantial real-time, each of the group of candidate live event devices having a camera configured to capture scenes from a physical environment and an antenna configured to transmit captured scenes over the distributed network;
associating a subset of the group of candidate live event devices with the live event based, at least in part, on the physical location of the live event and the received positions, the associated subset defining a group of live event devices;
generating an immersive representation of a physical environment of the live event using scenes captured from at least a first live event device and a second live event device, each of the group of live event devices;
generating a mixed reality environment by modifying the immersive visual representation of the physical environment with virtual element; and
causing a user device associated with the remote user to:
in a first mode, depict the mixed reality from a point of view of the first live event device; and
in a second mode, depict the mixed reality from a point of view of the second live event device.
20. A system for generating a mixed reality environment of a live event for a remote user, comprising:
a processing unit; and
a memory coupled to the processing unit, the memory for storing instruction which, when executed by the processing unit, causes the processing unit to:
determine a first mobile device is a live event device based, at least in part, on a position of the first mobile device during a programmed time window;
determine a second mobile device is a live event device based, at least in part, on a position of the second mobile device during the programmed time window;
receive scenes of a physical environment, the scenes captured by both the first mobile device and the second mobile device;
access a remote database having data elements that are characteristic of one or both of the live event or the remote user;
generate a mixed reality environment of the live event based on each of:
the scenes captured by the first mobile device;
the scenes captured by the second mobile device; and
the data elements of the remote database; and
transmit a signal including information of the mixed reality environment to a user device.