Mixed reality system for context-aware virtual object rendering
A computer-implemented method in conjunction with mixed reality gear (e.g., a headset) includes imaging a real scene encompassing a user wearing a mixed reality output apparatus. The method includes determining data describing a real context of the real scene, based on the imaging; for example, identifying or classifying objects, lighting, sound or persons in the scene. The method includes selecting a set of content including content enabling rendering of at least one virtual object from a content library, based on the data describing a real context, using various selection algorithms. The method includes rendering the virtual object in the mixed reality session by the mixed reality output apparatus, optionally based on the data describing a real context (“context parameters”). An apparatus is configured to perform the method using hardware, firmware, and/or software.
1. A method for operating a mixed reality session, the method comprising:
imaging, by one or more processors, a real scene for a user by a mixed reality output apparatus;
determining, by the one or more processors, context parameter data describing a real context of the imaged real scene;
selecting, by the one or more processors, a parameter set from a content library based on the context parameter data, wherein the parameter set enables a rendering of a virtual object in a mixed reality session of the mixed reality output apparatus, wherein selecting the parameter set includes selecting a class of virtual objects and randomly selecting an instance of the class of virtual objects as the virtual object; and
based on the parameter set, outputting, by the one or more processors, a rending of the virtual object in the mixed reality session by the mixed reality output apparatus.
2. The method of claim 1 , wherein imaging the real scene comprises:
analyzing, by the one or more processors, one or more images from an array based on photogrammetry, wherein the array includes at least one camera or at least one sensor; and
constructing, by the one or more processors, a three-dimensional model of the one or more images based on the analyzing of the one or more images from the array.
3. The method of claim 1 , wherein determining the context parameter data includes:
analyzing, by the one or more processors, one or more images of the real scene by utilizing an automatic object recognition algorithm or a scene recognition algorithm to determine the context parameter data.
4. The method of claim 1 , wherein selecting the parameter set is based on heuristic feedback.
5. The method of claim 4 , wherein the heuristic feedback includes implied user preference information or expressed user preference information corresponding to at least one past object rendered in the mixed reality session or a related session.
6. The method of claim 1 , wherein determining the context parameter data includes:
characterizing, by the one or more processors, a volumetric geometry of the real scene, wherein the volumetric geometry identifies one or more volume characteristics of one or more volumes within the real scene.
7. The method of claim 6 , wherein the one or more volume characteristics include a solid characteristic, a liquid characteristic, an air characteristic, or a mixed characteristic.
8. A computer system for operating a mixed reality session, the computer system comprising:
a memory having processor-readable instructions stored therein; and
one or more processors configured to access the memory and execute the processor-readable instructions, which when executed by the one or more processors configures the one or more processors to perform a plurality of functions, including functions for:
imaging, by the one or more processors, a real scene for a user by a mixed reality output apparatus;
determining, by the one or more processors, context parameter data describing a real context of the imaged real scene;
selecting, by the one or more processors, a parameter set from a content library based on the context parameter data, wherein the parameter set enables a rendering of a virtual object in a mixed reality session of the mixed reality output apparatus, wherein selecting the parameter set includes selecting a class of virtual objects and randomly selecting an instance of the class of virtual objects as the virtual object; and
based on the parameter set, outputting, by the one or more processors, a rending of the virtual object in the mixed reality session by the mixed reality output apparatus.
9. The computer system of claim 8 , wherein imaging the real scene comprises:
analyzing, by the one or more processors, one or more images from an array based on photogrammetry, wherein the array includes at least one camera or at least one sensor; and
constructing, by the one or more processors, a three-dimensional model of the one or more images based on the analyzing of the one or more images from the array.
10. The computer system of claim 8 , wherein determining the context parameter data includes:
analyzing, by the one or more processors, one or more images of the real scene by utilizing an automatic object recognition algorithm or a scene recognition algorithm to determine the context parameter data.
11. The computer system of claim 8 , wherein selecting the parameter set is based on heuristic feedback.
12. The computer system of claim 11 , wherein the heuristic feedback includes implied user preference information or expressed user preference information corresponding to at least one past object rendered in the mixed reality session or a related session.
13. The computer system of claim 8 , wherein determining the context parameter data includes:
characterizing, by the one or more processors, a volumetric geometry of the real scene, wherein the volumetric geometry identifies one or more volume characteristics of one or more volumes within the real scene.
14. The computer system of claim 13 , wherein the one or more volume characteristics include a solid characteristic, a liquid characteristic, an air characteristic, or a mixed characteristic.
15. A non-transitory computer-readable medium containing instructions for operating a mixed reality session, the instructions comprising:
imaging a real scene for a user by a mixed reality output apparatus;
determining context parameter data describing a real context of the imaged real scene;
selecting a parameter set from a content library based on the context parameter data, wherein the parameter set enables a rendering of a virtual object in a mixed reality session of the mixed reality output apparatus, wherein selecting the parameter set includes selecting a class of virtual objects and randomly selecting an instance of the class of virtual objects as the virtual object; and
based on the parameter set, outputting a rending of the virtual object in the mixed reality session by the mixed reality output apparatus.
16. The non-transitory computer-readable medium of claim 15 , wherein imaging the real scene comprises:
analyzing one or more images from an array based on photogrammetry, wherein the array includes at least one camera or at least one sensor; and
constructing a three-dimensional model of the one or more images based on the analyzing of the one or more images from the array.
17. The non-transitory computer-readable medium of claim 15 , wherein determining the context parameter data includes:
analyzing one or more images of the real scene by utilizing an automatic object recognition algorithm or a scene recognition algorithm to determine the context parameter data.
18. The non-transitory computer-readable medium of claim 15 , wherein selecting the parameter set is based on heuristic feedback.
19. The non-transitory computer-readable medium of claim 18 , wherein the heuristic feedback includes implied user preference information or expressed user preference information corresponding to at least one past object rendered in the mixed reality session or a related session.
20. The non-transitory computer-readable medium of claim 15 , wherein determining the context parameter data includes:
characterizing a volumetric geometry of the real scene, wherein the volumetric geometry identifies one or more volume characteristics of one or more volumes within the real scene.