Multi-user content sharing in immersive virtual reality environments
The technology disclosed relates to user interfaces for controlling augmented reality environments. Real and virtual objects can be seamlessly integrated to form an augmented reality by tracking motion of one or more real objects within view of a wearable sensor system using a combination a RGB (red, green, and blue) and IR (infrared) pixels of one or more cameras. It also relates to enabling multi-user collaboration and interaction in an immersive virtual environment. In particular, it relates to capturing different sceneries of a shared real world space from the perspective of multiple users. The technology disclosed further relates to sharing content between wearable sensor systems. In particular, it relates to capturing images and video streams from the perspective of a first user of a wearable sensor system and sending an augmented version of the captured images and video stream to a second user of the wearable sensor system.
1. A method of sharing content between wearable sensor systems, the method including:
capturing a first video stream of a first portion of a real world space using at least one camera electronically coupled to a first wearable sensor system;
generating, for display towards a first user, a first presentation output based on information from the first video stream and corresponding to the first portion of the real world space;
capturing a second video stream of a second portion of the real world space using at least one camera electronically coupled to the first wearable sensor system, wherein the first video stream and the second video stream correspond to a concurrent timespan;
sending, in association with a communication channel established between the first wearable sensor system and a second wearable sensor system, the second video stream to the second wearable sensor system; and
generating, for display towards a second user, a second presentation output based on information from the second video stream and corresponding to the second portion of the real world space.
2. The method of claim 1 , wherein a first set of features is detected corresponding to the first video stream and a second set of features is detected corresponding to the second video stream, the first set of features being processed to generate the first presentation output displayed towards the first user and the second set of features being processed to generate the second presentation output displayed towards the second user.
3. The method of claim 2 , wherein the second set of features further comprises one or more overlapping features in relation to the first set of features, and one or more nonoverlapping features in relation to the second set of features, the one or more nonoverlapping features relating to at least one of:
(i) a field of view corresponding to the second video stream,
(ii) a resolution of the second video stream,
(iii) a luminance measure of the second video stream,
(iv) an additional virtual graphic overlay to be combined with the second video stream, and
(v) an additional annotation overlay to be combined with the second video stream.
4. The method of claim 3 , wherein the first presentation output comprises a first virtual interface associated with a virtual reality system, and wherein the second presentation output comprises a second virtual interface associated with the virtual reality system.
5. The method of claim 4 ,
wherein at least one of the first feature set and the second feature set is processed, as input, to generate, as output, a three-dimensional map of at least one of the first portion and the second portion of the real world space, and
wherein the generated three-dimensional map is presented to a virtual reality system and the virtual reality system applies the generated three-dimensional map as a constraint upon which to construct a virtual reality environment.
6. The method of claim 5 ,
wherein the first virtual interface, responsive to a selection of an operation by the first user, enables the first user to interact with content within the virtual reality environment, and
wherein the second virtual interface, responsive to a selection of an operation by the second user, enables the second user to interact with content within the virtual reality environment.
7. The method of claim 6 , wherein the virtual reality environment enables collaboration between the first user and the second user participating in a collaborative operation, the collaborative operation facilitated by the first virtual interface and the second virtual interface, respective to the first user and the second user.
8. The method of claim 5 , wherein the virtual reality environment is configured to be updated to display one or more of new content and altered content, the updating of the display further comprising:
receiving, from the first wearable sensor system, an updated video stream of at least one of the first portion of the real world space and second portion of the real world space;
detecting, from the updated video stream, an updated set of features corresponding to the updated video stream;
processing, as input, at least one of the updated feature set and a previous three-dimensional map to generate, as output, an updated three-dimensional map; and
presenting, to the virtual reality system, the updated three-dimensional map to be applied as a constraint on the virtual reality environment.
9. The method of claim 1 , further comprising:
capturing a third video stream of a third portion of the real world space using at least one camera electronically coupled to the second wearable sensor system, wherein the first video stream and the third video stream correspond to a concurrent timespan;
generating, for display towards a second user, a third presentation output based on information from the third video stream and corresponding to the third portion of the real world space; and
sharing, in association with the communication channel established between the first wearable sensor system and the second wearable sensor system, data associated with the first presentation output and data associated with the third presentation output to at least one of the first wearable sensor system and the second wearable sensor system.
10. The method of claim 1 , wherein the first portion of the real world space and the second portion of the real world space overlap one another.
11. The method of claim 1 , wherein the first portion of the real world space and the second portion of the real world space do not overlap one another.
12. The method of claim 1 , wherein the first portion of the real world space is completely overlapped by the second portion of the real world space.
13. A non-transitory computer-readable recording medium having computer instructions recorded thereon for sharing content between wearable sensor systems, the computer instructions, when executed on one or more processors, causing the one or more processors to implement operations comprising:
capturing a first video stream of a first portion of a real world space using at least one camera electronically coupled to a first wearable sensor system;
generating, for display towards a first user, a first presentation output based on information from the first video stream and corresponding to the first portion of the real world space;
capturing a second video stream of a second portion of the real world space using at least one camera electronically coupled to the first wearable sensor system, wherein the first video stream and the second video stream correspond to a concurrent timespan;
sending, in association with a communication channel established between the first wearable sensor system and a second wearable sensor system, the second video stream to the second wearable sensor system; and
generating, for display towards a second user, a second presentation output based on information from the second video stream and corresponding to the second portion of the real world space.
14. The non-transitory computer-readable recording medium of claim 13 , wherein a first set of features is detected corresponding to the first video stream and a second set of features is detected corresponding to the second video stream, the first set of features being processed to generate the first presentation output displayed towards the first user and the second set of features being processed to generate the second presentation output displayed towards the second user.
15. The non-transitory computer-readable recording medium of claim 14 ,
wherein at least one of the first feature set and the second feature set is processed, as input, to generate, as output, a three-dimensional map of at least one of the first portion and the second portion of the real world space, and
wherein the generated three-dimensional map is presented to a virtual reality system and the virtual reality system applies the generated three-dimensional map as a constraint upon which to construct a virtual reality environment.
16. The non-transitory computer-readable recording medium of claim 15 , wherein the virtual reality environment enables collaboration between the first user and the second user participating in a collaborative operation, the collaborative operation facilitated by a first virtual interface and a second virtual interface, respective to the first user and the second user.
17. A system including one or more processors coupled to memory, the memory being loaded with computer instructions to share content between wearable sensor systems, the computer instructions, when executed on the one or more processors, causing the one or more processors to implement actions comprising:
capturing a first video stream of a first portion of a real world space using at least one camera electronically coupled to a first wearable sensor system;
generating, for display towards a first user, a first presentation output based on information from the first video stream and corresponding to the first portion of the real world space;
capturing a second video stream of a second portion of the real world space using at least one camera electronically coupled to the first wearable sensor system, wherein the first video stream and the second video stream correspond to a concurrent timespan;
sending, in association with a communication channel established between the first wearable sensor system and a second wearable sensor system, the second video stream to the second wearable sensor system; and
generating, for display towards a second user, a second presentation output based on information from the second video stream and corresponding to the second portion of the real world space.
18. The system of claim 17 , wherein:
a first set of features is detected corresponding to the first video stream and a second set of features is detected corresponding to the second video stream,
at least one of the first feature set and the second feature set is processed, as input, to generate, as output, a three-dimensional map of at least one of the first portion and the second portion of the real world space, and
the generated three-dimensional map is presented to a virtual reality system and the virtual reality system applies the generated three-dimensional map as a constraint upon which to construct a virtual reality environment.
19. The system of claim 18 , wherein the virtual reality environment is configured to be updated to display one or more of new content and altered content, the updating of the display further comprising:
receiving, from the first wearable sensor system, an updated video stream of at least one of the first portion of the real world space and second portion of the real world space;
detecting, from the updated video stream, an updated set of features corresponding to the updated video stream;
processing, as input, at least one of the updated feature set and a previous three-dimensional map to generate, as output, an updated three-dimensional map; and
presenting, to the virtual reality system, the updated three-dimensional map to be applied as a constraint on the virtual reality environment.
20. The system of claim 19 , further comprising:
capturing a third video stream of a third portion of the real world space using at least one camera electronically coupled to the second wearable sensor system, wherein the first video stream and the third video stream correspond to a concurrent timespan;
generating, for display towards a second user, a third presentation output based on information from the third video stream and corresponding to the third portion of the real world space; and
sharing, in association with the communication channel established between the first wearable sensor system and the second wearable sensor system, data associated with the first presentation output and data associated with the third presentation output to at least one of the first wearable sensor system and the second wearable sensor system.